Utility to parse HTTP response
Hi all, To build HTTP request, you use 'curl'. After you get HTTP response, what utility do you use to extract stuffs you want? JSON format has 'jq', and XML format has 'xmlstarlet'. I'm looking for something like that but for HTTP format. --
On Fri, 31 Jan 2025 at 01:07, William Park via talk<talk@gtalug.org> wrote:
After you get HTTP response, what utility do you use to extract stuffs you want? In rough order
* XML tools like xslt * http parsers like https://github.com/nodejs/llhttp (it says nodejs, but this is in c) * Normal code that uses a definition in BNF. I used to use RFC 2616, but that was back in 1999. --dave -- David Collier-Brown, | Always do right. This will gratify System Programmer and Author | some people and astonish the rest davecb@spamcop.net | -- Mark Twain
On Fri, Jan 31, 2025 at 01:07:18AM -0500, William Park via talk wrote:
Hi all,
To build HTTP request, you use 'curl'.
After you get HTTP response, what utility do you use to extract stuffs you want? JSON format has 'jq', and XML format has 'xmlstarlet'. I'm looking for something like that but for HTTP format.
Couldn't it be just about anything? After all you can transfer lots of file formats using http and it would depend on the content type what tool makes sense to perse the response. -- Len Sorensen
Yes, it can be anything because HTTP has 3 parts really... first line, headers, and body. But, I'm tired of doing it again and again. Also, multiple parts and multiple encoding methods are hassle to keep track. On 2025-01-31 12:11, Lennart Sorensen wrote:
On Fri, Jan 31, 2025 at 01:07:18AM -0500, William Park via talk wrote:
Hi all,
To build HTTP request, you use 'curl'.
After you get HTTP response, what utility do you use to extract stuffs you want? JSON format has 'jq', and XML format has 'xmlstarlet'. I'm looking for something like that but for HTTP format.
Couldn't it be just about anything? After all you can transfer lots of file formats using http and it would depend on the content type what tool makes sense to perse the response.
On Fri, Jan 31, 2025 at 07:03:41PM -0500, William Park via talk wrote:
Yes, it can be anything because HTTP has 3 parts really... first line, headers, and body. But, I'm tired of doing it again and again. Also, multiple parts and multiple encoding methods are hassle to keep track.
Well certainly with curl you can dump the headers to a file and the content to another file and could then use the content-type line in the header to determine what type the content is. I don't recall multipart being used for received data only for uploads, but maybe that is just because I haven't looked hard enough. A quick search seems to indicate that the received data is always just one file. Of course if you do persistent connections you get to do multiple requests and multiple receives (each with their own header) on the same socket, but that doesn't really change anything. -- Len Sorensen
Found a solution from KWLUG... It seems 'curl' has --write-out feature, similar to printf() format, or here-document feature of shell. Eg. curl -o x.html -w '%output{x.json}%{http_code}\n%{http_version}\n%{header_json}\n%{json}\n' https://www.gtalug.org/ On 2025-01-31 01:07, William Park via talk wrote:
Hi all,
To build HTTP request, you use 'curl'.
After you get HTTP response, what utility do you use to extract stuffs you want? JSON format has 'jq', and XML format has 'xmlstarlet'. I'm looking for something like that but for HTTP format.
participants (4)
-
David Collier-Brown -
Lennart Sorensen -
Scott Allen -
William Park