Help need in bash
I want to build a script to convert any *.png files to *.jpg files using convert. Doing manually works fine. Building a script has presented problem. I am well versed in programming but a novice with bash. My loop through the files works fine. #!/bin/bash cd /big1/memes/; shopt -s nullglob for f in "*.png" do echo $f done I get a list of filenames with an extension of .png But when I try to get the basename with: #!/bin/bash cd /big1/memes/; shopt -s nullglob for f in "*.png" do echo $f b=${f%.png} echo $b done I first get a list of the .png files, then a list of all files, both the existing .jog and the .png. This gets a WTF reaction from me. Can someone explain and help me get closer to getting this to work? Thanks -- Stephen
On Thu, May 31, 2018, 14:40 Stephen via talk, <talk@gtalug.org> wrote:
#!/bin/bash cd /big1/memes/;
shopt -s nullglob for f in "*.png" do echo $f
This is where you get the PNG echoes. b=${f%.png}
echo $b done
You can also do convert file.{png,jpg} and bash will do the magic. What I don't know is if that will accept a glob. Using convert is okay, but you won't end up with the smallest JPEGs that way. Stewart
| From: Stewart Russell via talk <talk@gtalug.org> | Using convert is okay, but you won't end up with the smallest JPEGs that | way. What do you recommend? (Lightning talk?)
On Jun 1, 2018, at 10:34, D. Hugh Redelmeier via talk <talk@gtalug.org> wrote:
| From: Stewart Russell via talk <talk@gtalug.org>
| Using convert is okay, but you won't end up with the smallest JPEGs that | way.
What do you recommend? (Lightning talk?)
I would recommend being aware of the “-quality” flag for convert (it tends to default to 92). Valid values for it range between 1 (smallest file, worst artefacts) and 100 (largest file, best quality). Give it a try with different setting, pick the one that is the best compromise. Use some files that JPEG is both bad and good at compressing in your test set. Another flag that is of potential interest is “-strip”, which removes all non-required metadata. It’s a bit of a double-edged sword, as part of that removed data is the ICC profile. The embedded colour profile is what gives the viewing application a reference for the colours. No reference means the file may be rendered with the colours slightly off. Now if this is purely for web use, we’re at the point that sRGB is assumed. If you first ensure that your source data is correct in sRGB, you can strip off the profile and web browsers will display it correctly. In smaller, thumbnail-sized images, the metadata can exceed the size of the image data. Also removed by “-strip”, would be the “non-destructive” edits performed by some applications. These “edits” are a list of modifications to the image shoved into the XMP metadata, and in bad cases, I’ve seen that list of XML exceed 5MB and bring ExifTool to a screeching halt. I don’t feel as bad about removing that cruft. If you use non-destructive editing tools, be sure to save a flat copy somewhere and run your outside tools against the copy. Seneca
On 2018-06-01 10:34 AM, D. Hugh Redelmeier via talk wrote:
| Using convert is okay, but you won't end up with the smallest JPEGs that | way.
What do you recommend? (Lightning talk?)
oh I wish - the next couple of weeks I'm running more events than is sensible. anyway, of course I've overthunk this, so some notes in lieu: * cjpeg - the IJG/libjpeg-turbo command offers easier access to JPEG options than IM/GM's convert, such as greyscale * you can very often have a much lower JPEG quality than defaults, but keep the originals somewhere safe 'cos JPEG is lossy * don't use arithmetic-coding compressed JPEGs: the browser support is nil * (almost all) JPEGs are made up of 8×8 px colour approximation cells. If your image isn't an exact multiple of 8 px in each direction, you're going to waste space in the file * consequently, cropping a JPEG can do unexpected bad things to the output. jpegtran — another IJG/libjpeg-turbo tool — is one of the very few programs that can do a true lossless crop of a JPEG file * JPEGs are for continuous-colour photographic images, and any sharp change in colour — like you'd get in computer-generated or line graphics — causes weird colour fringes / artifacts around the edges. The "modern" way to deal with this is to rack the JPEG quality up to 95%+ and live with huge files. The "right" way to deal with this is to add a *tiny* bit of colour noise to the image, and it will kill the fringes off. (JPEGs essentially store the result of signal processing the source image rather than the image itself. If you've ever played with signals (analogue or digital), sharp edges in any input waveform can cause "filter ringing" or unwanted harmonics in the output. JPEG fringes are unwanted filter harmonics, basically) * BUT … nobody really cares (or needs to) about JPEG size these days as every webpage is megabytes of crap anyway. * almost nobody (outside specialist domains) uses the replacement for the JPEG format, JPEG-2000. This is a shame, as it has some very nifty features like progressive viewing (so an overview will only load a small part of the file) and truly impressively tiny file sizes. JPEG-2000 takes a fair amount of processing power to render (deemed too much/too slow for the web circa 2001) and was until recently encumbered by some annoying patents. Ubuntu deals with this very badly by patching a bunch of tools (like ImageMagick) such that when you ask for JPEG-2000, it gives you a JPEG. This is bad and wrong and they must get off my lawn forthwith. cheers, Stewart (Oh, and if you've made it this far: I see that local(ish) Linux-friend Bob Jonkman is standing as a Green candidate in Kitchener-Conestoga: https://gpo.ca/candidate/bob-jonkman/)
On 2018-06-01 10:34 AM, D. Hugh Redelmeier via talk wrote: * cjpeg - the IJG/libjpeg-turbo command offers easier access to JPEG options than IM/GM's convert, such as greyscale
* you can very often have a much lower JPEG quality than defaults, but keep the originals somewhere safe 'cos JPEG is lossy
* don't use arithmetic-coding compressed JPEGs: the browser support is nil
* (almost all) JPEGs are made up of 8×8 px colour approximation cells. If your image isn't an exact multiple of 8 px in each direction, you're going to waste space in the file
* consequently, cropping a JPEG can do unexpected bad things to the output. jpegtran — another IJG/libjpeg-turbo tool — is one of the very few programs that can do a true lossless crop of a JPEG file
* JPEGs are for continuous-colour photographic images, and any sharp change in colour — like you'd get in computer-generated or line graphics — causes weird colour fringes / artifacts around the edges. The "modern" way to deal with this is to rack the JPEG quality up to 95%+ and live with huge files. The "right" way to deal with this is to add a *tiny* bit of colour noise to the image, and it will kill the fringes off. (JPEGs essentially store the result of signal processing the source image rather than the image itself. If you've ever played with signals (analogue or digital), sharp edges in any input waveform can cause "filter ringing" or unwanted harmonics in the output. JPEG fringes are unwanted filter harmonics, basically)
* BUT … nobody really cares (or needs to) about JPEG size these days as every webpage is megabytes of crap anyway.
* almost nobody (outside specialist domains) uses the replacement for the JPEG format, JPEG-2000. This is a shame, as it has some very nifty features like progressive viewing (so an overview will only load a small part of the file) and truly impressively tiny file sizes. JPEG-2000 takes a fair amount of processing power to render (deemed too much/too slow for the web circa 2001) and was until recently encumbered by some annoying patents. Ubuntu deals with this very badly by patching a bunch of tools (like ImageMagick) such that when you ask for JPEG-2000, it gives you a JPEG. This is bad and wrong and they must get off my lawn forthwith.
Hugh, you reminded of this presentation I watched a few days ago. < https://youtu.be/hQZ7Xg7q7zw>. Highly recommended. Regards, Clifford Ilkay +1 647-778-8696
convert file.{png,jpg}is same as convert file.png file.jpgAnd, convert *.{png,jpg}is same as convert *.png *.jpg Sent from Yahoo Mail on Android On Fri, Jun 1, 2018 at 12:32 PM, Stewart C. Russell via talk<talk@gtalug.org> wrote: On 2018-05-31 03:00 PM, Stewart Russell wrote:
convert file.{png,jpg}
and bash will do the magic. What I don't know is if that will accept a glob.
narrator voice: it didn't accept a glob. (that is, convert *.{png,jpg} does *something*, but neither what you'd expect nor want) --- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
On 2018-05-31 03:00 PM, Stewart Russell wrote:
convert *.{png,jpg}
does *something*, but neither what you'd expect nor want)
Multics had an "equal name" feature that conspired with the star-name parser to do what you would expect. A program like convert could have used it like convert *.png =.jpg Name components that were matched by '*' could be retrieved to replace '=' in building a new file name. It probably only worked because star-names weren't expanded by the command shell. Each command program parsed its own command line (with heavy use of standard subroutines.) That way, the program had control over what command line arguments were being matched together.
| From: Stewart C. Russell via talk <talk@gtalug.org> | On 2018-05-31 03:00 PM, Stewart Russell wrote: | > | > convert file.{png,jpg} | > | > and bash will do the magic. What I don't know is if that will accept a | > glob. | | narrator voice: it didn't accept a glob. | | (that is, | | convert *.{png,jpg} | | does *something*, but neither what you'd expect nor want) Image Magick's convert command sure is magic. But its syntax does not follow UNIX conventions. SYNOPSIS convert [input-option] input-file [output-option] output-file "accept a glob" is not a UNIX thing. globbing is done by the shell. What you mean is: accepts a sequence of pathnames on the command line, often denoted "file..." in a synopsis. Like this SYNOPSIS cat [OPTION]... [FILE]... The convert synopsis is actually wrong, I think. I think that it should be convert [input-option]... input-file [output-option]... output-file What you wanted to do is actually more awkward in UNIX than in systems where the globbing is done by the program (I think). In those systems, the "filename extension" is a first-class notion. for i in *.png ; do convert "$i" "$(basename "$i" .png).jpg" ; done I don't even understand why the inner quotes don't need to be escaped. I tested it with this simulation: for i in "hello.png" "good bye.png" ; do echo convert "$i" "$(basename "$i" .png).jpg" ; done basename strips any directory prefix off of its argument but I didn't worry because *.png won't produce paths with directory prefixes. Beware if you try to generalize this code. bash has powerful but odd macro substitution operators. Here's a version of the code where the basename has been replaced by one of these operator. I have no idea if this works in other shells for i in *.png ; do convert "$i" "${i%.png}.jpg" ; done (I learned the Bourne shell in the 1970's and usually stick to that original simple language. I have to read bash(1) to find things like this.)
On May 31, 2018, at 14:40, Stephen via talk <talk@gtalug.org> wrote:
I want to build a script to convert any *.png files to *.jpg files using convert.
Doing manually works fine.
Building a script has presented problem. I am well versed in programming but a novice with bash.
My loop through the files works fine.
#!/bin/bash cd /big1/memes/;
shopt -s nullglob for f in "*.png" do echo $f done
I get a list of filenames with an extension of .png
In this case, the value of $f is ‘*.png’. Your loop iterates once, and the list of PNG files is generated as a shell expansion before evaluating the “echo”.
But when I try to get the basename with:
#!/bin/bash cd /big1/memes/;
shopt -s nullglob for f in "*.png" do echo $f b=${f%.png} echo $b done
I first get a list of the .png files, then a list of all files, both the existing .jog and the .png. This gets a WTF reaction from me.
Once more, $f is ‘*.png’ and is being expanded for the execution of the echo. Your brace expansion looks to be converting ‘*.png’ to the string ‘*’, and assigning that to b. The when evaluating the line “echo $b”, the value of $b, ‘*’, is expanded prior to execution.
Can someone explain and help me get closer to getting this to work?
You were a touch sloppy with your quoting. You should have left “*.png” unquoted in the for loop so that the list of file names to process would expand there, and stuck $f and $b in double quotes so their values would not expand. Seneca
That is what I needed. Thank you! On 2018-05-31 04:47 PM, Seneca Cunningham via talk wrote:
On May 31, 2018, at 14:40, Stephen via talk <talk@gtalug.org> wrote:
I want to build a script to convert any *.png files to *.jpg files using convert.
Doing manually works fine.
Building a script has presented problem. I am well versed in programming but a novice with bash.
My loop through the files works fine.
#!/bin/bash cd /big1/memes/;
shopt -s nullglob for f in "*.png" do echo $f done
I get a list of filenames with an extension of .png
In this case, the value of $f is ‘*.png’. Your loop iterates once, and the list of PNG files is generated as a shell expansion before evaluating the “echo”.
But when I try to get the basename with:
#!/bin/bash cd /big1/memes/;
shopt -s nullglob for f in "*.png" do echo $f b=${f%.png} echo $b done
I first get a list of the .png files, then a list of all files, both the existing .jog and the .png. This gets a WTF reaction from me.
Once more, $f is ‘*.png’ and is being expanded for the execution of the echo. Your brace expansion looks to be converting ‘*.png’ to the string ‘*’, and assigning that to b. The when evaluating the line “echo $b”, the value of $b, ‘*’, is expanded prior to execution.
Can someone explain and help me get closer to getting this to work?
You were a touch sloppy with your quoting. You should have left “*.png” unquoted in the for loop so that the list of file names to process would expand there, and stuck $f and $b in double quotes so their values would not expand.
Seneca --- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
-- Stephen
| From: Seneca Cunningham via talk <talk@gtalug.org> | You were a touch sloppy with your quoting. Rules for quoting in bash (or any language with macro semantics) are hard. Quoting doesn't seem to be natural for humans. Without quotes, data can bleed into code. When I'm writing a bash script, I try to quote every macro use except for those that must not be quoted. The reason: without quoting, the use will probably break if there is a space (or other odd thing) in the value but "normal" testing rarely uncovers this. UNIX/Linux folks don't think that spaces belong in pathnames but GUIs and the OS don't agree. Repeat advice: - use "set -u" to turn references to undefined macros (spelling errors) into bash errors - use "set -e" so that a shell script stops when a command returns an error value that the script tacitly ignores. bash scripts are easy to get wrong. I've found that these habits expose or prevent a significant number of mistakes. They also remove these issues from consideration when you are trying to puzzle out some bash problem.
On Fri, Jun 01, 2018 at 10:56:39AM -0400, D. Hugh Redelmeier via talk wrote:
Rules for quoting in bash (or any language with macro semantics) are hard. Quoting doesn't seem to be natural for humans. Without quotes, data can bleed into code.
When I'm writing a bash script, I try to quote every macro use except for those that must not be quoted.
The reason: without quoting, the use will probably break if there is a space (or other odd thing) in the value but "normal" testing rarely uncovers this.
UNIX/Linux folks don't think that spaces belong in pathnames but GUIs and the OS don't agree.
Repeat advice:
- use "set -u" to turn references to undefined macros (spelling errors) into bash errors
- use "set -e" so that a shell script stops when a command returns an error value that the script tacitly ignores.
bash scripts are easy to get wrong. I've found that these habits expose or prevent a significant number of mistakes. They also remove these issues from consideration when you are trying to puzzle out some bash problem.
Oh yes quoting is fun. $ ls 'a b c.png' a.png b.png c.png $ for f in "*.png"; do echo "$f"; done a b c.png a.png b.png c.png $ for f in "*.png"; do echo "$f"; done *.png $ for f in *.png; do echo "$f"; done a b c.png a.png b.png c.png $ for f in *.png; do ls $f; done ls: cannot access 'a': No such file or directory ls: cannot access 'b': No such file or directory c.png a.png b.png c.png Some places you must use it and some places you must not. -- Len Sorensen
| From: Lennart Sorensen via talk <talk@gtalug.org> | $ for f in "*.png"; do echo "$f"; done | a b c.png a.png b.png c.png I think that you mean ... echo $f ...
On Fri, Jun 01, 2018 at 01:09:28PM -0400, D. Hugh Redelmeier via talk wrote:
| From: Lennart Sorensen via talk <talk@gtalug.org>
| $ for f in "*.png"; do echo "$f"; done | a b c.png a.png b.png c.png
I think that you mean ... echo $f ...
Oh maybe I cut and pasted the wrong test line. :) -- Len Sorensen
participants (9)
-
Clifford Ilkay -
D. Hugh Redelmeier -
lsorense＠csclub.uwaterloo.ca -
mwilson＠Vex.Net -
Seneca Cunningham -
Stephen -
Stewart C. Russell -
Stewart Russell
-
William Park