On Mon, Jan 08, 2018 at 04:48:13PM -0500, Andrew Paolucci via talk wrote:

I believe from what is bouncing around the internet the upcoming line of Intel Processors this year will still have the physical bug and the software patch is seen as the "fix" for the foreseeable future. I was hoping for a refund/return program similar to the Pentium floating point bug. If you are willing to keep supporting Intel after this snafu I don't see a point in waiting unless it's for new tech, not secure tech. AMD is a solid option moving forward though, no hidden network stack or Minx OS onboard.

I think a big difference is that when the FDIV issue was found, the pentium was still a current in production CPU. Swapping them out was an option. I am pretty sure intel isn't making Pentium Pro processors on a 350nm process anymore. -- Len Sorensen

| From: Clifford Ilkay via talk <talk@gtalug.org> | By the way, I don't see the point of doing a new build with a CPU that is | four generations old and with only 16GB that ends up costing $1600+. You | could purchase a refurbished, off-lease workstation of that vintage for | about a quarter of the price with more RAM. It would not likely have an SSD | but you could always add one if you really wanted to. I really agree with this. Here, for example, is a reasonable off-lease "business class" computer from a reputable dealer: <https://www.dellrefurbished.ca/desktop-computers?filter_memory=79&filter_processor_brand=357> $429 for an i5-4570 with 16G of RAM <https://www.dellrefurbished.ca/desktop-computers?filter_processor_brand=357> a whole bunch to chose from, with most with less RAM and lower prices. Note: Dell Refurbished have a sale times a year with significant discounts. Perhaps three times since you configured your system on PC partspicker. Their stock comes and goes. Right now is probably a low considering the Boxing Day sale they had. If you are daring, knowledgeable, and patient, there are better deal on Kijiji or ebay, for example. For example, the box I'm typing this on came from Staples via ebay. I paid about $650 (including taxes and shipping) for a refurb Haswell i7 system with 12G of RAM and a discrete graphics card. This was 3.5 years ago.

| From: Russell via talk <talk@gtalug.org> | On January 8, 2018 10:33:00 AM EST, "Steve Petrie, P.Eng. via talk" <talk@gtalug.org> wrote: | >Intel 4-Core i5-4460 3.2GHz Haswell Processor. (I know, I know, the | >Intel Haswell CPU series is obsolete, but let's just ignore this fact | >for the present discussion.) | | I personally wouldn't say its obsolete until there is a significant lack | of replacement parts or, essential bios functionality requirements for | running the software are unavailable in updates. Haswell machines are fine as anything now but - they use older memory which is already starting to fade from the scene - brand new Haswell chips are not a bargain compared with current chips - firmware updates have already disappeared for many Haswell motherboards (like the one I'm typing on) | Although Spectre and Meltdown may make it look like this is the case, I | am reminded of the immortal words of Thomas Edison who said, paraphrased | - nothing good ever just works, it needs working on. I'm not quite up to speed on the various horrible problems that have come out recently. So take what I say with a grain of salt. - Meltdown is Intel-only. I don't see how it can be fixed on an existing chip -- I don't see microcode updates doing that. OS-level mitigations should work, at the cost of performance. There is a feature of recent Intel CPUs called PCID (Process Context IDentifiers) that might help reduce the performance cost but it isn't yet exploited. PCID is probably in Haswell, but I don't know. - Spectre is a general problem in modern high-performance processors. Luckily, there are probably only a few places that need to be changed to avoid the attack. In general terms, interpreters that are the bulwark against untrusted code need to be hardened. Think: java, javascript, eBPF. - separately, there are horrible security holes in Intel's and AMD's chipsets. These cannot be mitigated by anyone but the manufacturer because the holes are in secret and proprietary features. Any fix must come in the form of either mitigating rituals specified by the manufacturer or in firmware updates. Processor firmware updates installed by Linux cannot do the job completely because these flaws are in code that is active before the OS is loaded. Thus these processor firmware fixes must be embedded in motherboard firmware updates (mistakenly called BIOS updates). So: you want to have a system with a motherboard which will still get firmware updates. This is a really annoying requirement since motherboard firmware updates often cease rather quickly. The main exception is in systems intended for business or server use. For example, in my experience, ThinkPads get firmware updates for many years but other Lenovo laptops get perhaps a year of firmware updates. | My choices were based on the fact that the Z370 was designed with only | forward compatibility in mind and i-7x was a mature 14nm manufacturing | technology. If you are going to the trouble and expense of assembling a system yourself, this seems like the right attitude. | >b.. 2. Invest a week or two to investigate AMD CPU options that will | >avoid altogether the Intel Meltdown bug. This strategy could lead to a | >lengthy necessary respecification of other components in the PC build | >spec, depending on how an AMD architecture fits with peripherals. | >An ARM CPU is not an option, as one of the other operating system I | >plan to run on the new PC is DragonFlyBSD, which only runs on Intel and | >AMD processors (not ARM). | | Waiting won't necessarily hurt, but it won't necessarily change | anything. As consumers, we believe what we are told by experts about | zero day exploits, this is one of those days. | | Certainly this is industry wide and not just within Intel. There are | perhaps a couple of singular exceptions to-date. Yes. The IC processes are so effective that complexity is cheap. It is really hard to make a complex system secure. This affects all processor manufacturers. | Having a predictive | microcode hypervisor has been described as having a RISC backend with a | CISC frontend. I don't think that you are using the word "hypervisor" correctly. I also don't see how "predictive" belongs in this sentence. | This issue could probably be mitigated by switching | entirely to RISC. I don't know what you mean by "this issue". I do think that RISC is simpler and should have fewer cases to get right. But real-world RISC architectures seem to get more complicated at quite a pace. | >And then there is the Spectre bug also lurking ... | | Microsoft has stopped, temporarily, updating AMD due to poor documentation. | | https://www.theverge.com/2018/1/9/16867068/microsoft-meltdown-spectre-security-updates-amd-pcs-issues> Thanks for this reference. I hadn't seen that. Microsoft isn't too clear about their finger-pointing (that doesn't mean that they are wrong). This notice isn't very clear: <https://support.microsoft.com/en-us/help/4073707/windows-os-security-update-block-for-some-amd-based-devices> - it would be really handy if they explained what "some" means. Which AMD-based devices are currently blocked - this run-on sentence might not mean what it is supposed to After investigating, Microsoft has determined that some AMD chipsets do not conform to the documentation previously provided to Microsoft to develop the Windows operating system mitigations to protect against the chipset vulnerabilities known as Spectre and Meltdown. - Spectre and Meltdown are different. The fixes are different. Preventing installation of the fixes for both is probably a mistake. To be honest, I don't understand why these mitigations involve a chipset at all. | As end users we are all beholding to the good faith disclosure of | manufacturing industries. Some early reports say that only a revamp of | CPU design will fully address Spectre. Actually, we probably want speculative execution in a way that makes this leak possible. We just want a simple code sequence that can stop this speculation for some particular references. Then we have to trust the software people to use that sequence where it is needed. | Here's an interesting link to the KASLR report on address space layout | randomization and problematic Intsructional Level Parallelism. | | https://www.google.ca/url?sa=t&source=web&rct=j&url=https://gruss.cc/files/kaiser.pdf&ved=2ahUKEwjOtpWZ-srYAhUE9YMKHT-yDqUQFjAEegQIARAB&usg=AOvVaw0vUvGParJ9_uWXs2iph7KZ That's a very indirect URL. How about <https://gruss.cc/files/kaiser.pdf> | I'd be much more worried if this had been discovered running in the wild | rather than by lab testing and research, but I've been wrong before. Everyone is. But a lot of the bad players try to use these things in ways that won't be detected. | | >Thanks in advance, | | Good luck with your choices. Remember, you have to be compromised in the | first place for these exploits to take hold. Not true. Spectre can be exploited via javascript on a web site (including an ad from who knows where). | In the weeks before this broke and while I was configuring Fedora 27 on | this new HW, I updated based on notices of important changes. | | No new updates since this news broke, so I did a refresh update. That | produced no noticable changes. Mind you that's just ad hoc observations | from the system monitor and not benchmarked results. YMMV There have been several relevant Fedora 27 updates. Some were released just as the first inklings were appearing. In fact, the first news broke because an AMD patch to the kernel spilled the beans. The AMD patch turned off the already present mitigation for Meltdown in the case of AMD processors.

| From: David Collier-Brown via talk <talk@gtalug.org> | On 09/01/18 12:23 PM, D. Hugh Redelmeier via talk wrote: | > In fact, the first news broke because an AMD patch to the kernel | > spilled the beans. The AMD patch turned off the already present | > mitigation for Meltdown in the case of AMD processors. | > - | | Erk!  Can you point us to a description this? It seem slightly pessimal (;-)) I think that this posting to the Linux Kernel Mailing List is considered to have let the cat out of the bag for Meltdown. <https://lkml.org/lkml/2017/12/27/2> The problem had been "responsibly disclosed" about half a year before to folks considered responsible (i.e. not us). I'm not 100% comfortable with "responsible disclosure". Especially with half a year lead time.

On January 9, 2018 12:23:24 PM EST, "D. Hugh Redelmeier via talk" <talk@gtalug.org> wrote:

| From: Russell via talk <talk@gtalug.org>

| On January 8, 2018 10:33:00 AM EST, "Steve Petrie, P.Eng. via talk" <talk@gtalug.org> wrote:

| >Intel 4-Core i5-4460 3.2GHz Haswell Processor. (I know, I know, the | >Intel Haswell CPU series is obsolete, but let's just ignore this fact | >for the present discussion.) | | I personally wouldn't say its obsolete until there is a significant lack | of replacement parts or, essential bios functionality requirements for | running the software are unavailable in updates.

Haswell machines are fine as anything now but

- they use older memory which is already starting to fade from the scene

- brand new Haswell chips are not a bargain compared with current chips

- firmware updates have already disappeared for many Haswell motherboards (like the one I'm typing on)

| Although Spectre and Meltdown may make it look like this is the case, I | am reminded of the immortal words of Thomas Edison who said, paraphrased | - nothing good ever just works, it needs working on.

I'm not quite up to speed on the various horrible problems that have come out recently. So take what I say with a grain of salt.

- Meltdown is Intel-only. I don't see how it can be fixed on an existing chip -- I don't see microcode updates doing that. OS-level mitigations should work, at the cost of performance. There is a feature of recent Intel CPUs called PCID (Process Context IDentifiers) that might help reduce the performance cost but it isn't yet exploited. PCID is probably in Haswell, but I don't know.

- Spectre is a general problem in modern high-performance processors. Luckily, there are probably only a few places that need to be changed to avoid the attack. In general terms, interpreters that are the bulwark against untrusted code need to be hardened. Think: java, javascript, eBPF.

- separately, there are horrible security holes in Intel's and AMD's chipsets. These cannot be mitigated by anyone but the manufacturer because the holes are in secret and proprietary features. Any fix must come in the form of either mitigating rituals specified by the manufacturer or in firmware updates.

Processor firmware updates installed by Linux cannot do the job completely because these flaws are in code that is active before the OS is loaded.

Thus these processor firmware fixes must be embedded in motherboard firmware updates (mistakenly called BIOS updates).

So: you want to have a system with a motherboard which will still get firmware updates. This is a really annoying requirement since motherboard firmware updates often cease rather quickly. The main exception is in systems intended for business or server use.

For example, in my experience, ThinkPads get firmware updates for many years but other Lenovo laptops get perhaps a year of firmware updates.

| My choices were based on the fact that the Z370 was designed with only | forward compatibility in mind and i-7x was a mature 14nm manufacturing | technology.

If you are going to the trouble and expense of assembling a system yourself, this seems like the right attitude.

Well your and Lennarts suggestions helped to move me to a more correct value point choice for my situation. Especially your CPU recommendation; dodged a bit of shrapnel with that one, at least according to some of the benchmarks I'm seeing. I checked for vulnerabilities. wget https://raw.githubusercontent.com/speed47/spectre-meltdown-checker/master/sp... Ran it and then updated I the MB firmware to the most recent v606. The checker now says. CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1' * Checking count of LFENCE opcodes in kernel: NO (only 34 opcodes found, should be >= 70)

STATUS: VULNERABLE (heuristic to be improved when official patches become available)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2' * Mitigation 1 * Hardware (CPU microcode) support for mitigation: YES * Kernel support for IBRS: NO * IBRS enabled for Kernel space: NO * IBRS enabled for User space: NO * Mitigation 2 * Kernel compiled with retpoline option: NO Internet trampoline/java fix needed? Two choices to follow. * Kernel compiled with a retpoline-aware compiler: NO

STATUS: VULNERABLE (IBRS hardware + kernel support OR kernel with retpoline are needed to mitigate the vulnerability)

One down in any case. CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3' * Kernel supports Page Table Isolation (PTI): YES * PTI enabled and active: YES

STATUS: NOT VULNERABLE (PTI mitigates the vulnerability)

| >b.. 2. Invest a week or two to investigate AMD CPU options that will | >avoid altogether the Intel Meltdown bug. This strategy could lead to a | >lengthy necessary respecification of other components in the PC build | >spec, depending on how an AMD architecture fits with peripherals. | >An ARM CPU is not an option, as one of the other operating system I | >plan to run on the new PC is DragonFlyBSD, which only runs on Intel and | >AMD processors (not ARM). | | Waiting won't necessarily hurt, but it won't necessarily change | anything. As consumers, we believe what we are told by experts about | zero day exploits, this is one of those days. | | Certainly this is industry wide and not just within Intel. There are | perhaps a couple of singular exceptions to-date.

Yes.

The IC processes are so effective that complexity is cheap. It is really hard to make a complex system secure. This affects all processor manufacturers.

| Having a predictive | microcode hypervisor has been described as having a RISC backend with a | CISC frontend.

I don't think that you are using the word "hypervisor" correctly. I also don't see how "predictive" belongs in this sentence.

| This issue could probably be mitigated by switching | entirely to RISC.

I don't know what you mean by "this issue".

I do think that RISC is simpler and should have fewer cases to get right. But real-world RISC architectures seem to get more complicated at quite a pace.

| >And then there is the Spectre bug also lurking ... | | Microsoft has stopped, temporarily, updating AMD due to poor documentation. | | https://www.theverge.com/2018/1/9/16867068/microsoft-meltdown-spectre-security-updates-amd-pcs-issues>

Thanks for this reference. I hadn't seen that.

Microsoft isn't too clear about their finger-pointing (that doesn't mean that they are wrong). This notice isn't very clear: <https://support.microsoft.com/en-us/help/4073707/windows-os-security-update-block-for-some-amd-based-devices>

- it would be really handy if they explained what "some" means. Which AMD-based devices are currently blocked

- this run-on sentence might not mean what it is supposed to

After investigating, Microsoft has determined that some AMD chipsets do not conform to the documentation previously provided to Microsoft to develop the Windows operating system mitigations to protect against the chipset vulnerabilities known as Spectre and Meltdown.

- Spectre and Meltdown are different. The fixes are different. Preventing installation of the fixes for both is probably a mistake.

To be honest, I don't understand why these mitigations involve a chipset at all.

I'm asking myself the same question.

| As end users we are all beholding to the good faith disclosure of | manufacturing industries. Some early reports say that only a revamp of | CPU design will fully address Spectre.

Actually, we probably want speculative execution in a way that makes this leak possible. We just want a

simple code sequence that can stop

this speculation for some particular references. Then we have to trust the software people to use that sequence where it is needed.

I'm in 100 per cent agreement with you there. Try and tell a merchant class that security by obscurity, plainly and simply, does not work; see how far that gets you. Having said that, as I become more familiar with policy management tools, I become more trusting of them. Notwithstanding that the it was NSA which produced SElinux.

| Here's an interesting link to the KASLR report on address space layout | randomization and problematic Intsructional Level Parallelism. | | https://www.google.ca/url?sa=t&source=web&rct=j&url=https://gruss.cc/files/kaiser.pdf&ved=2ahUKEwjOtpWZ-srYAhUE9YMKHT-yDqUQFjAEegQIARAB&usg=AOvVaw0vUvGParJ9_uWXs2iph7KZ

That's a very indirect URL. How about <https://gruss.cc/files/kaiser.pdf>

Sorry should have trimmed that url. It's an interesting read considering the situation my current setup choices.

| I'd be much more worried if this had been discovered running in the wild | rather than by lab testing and research, but I've been wrong before.

Everyone is. But a lot of the bad players try to use these things in ways that won't be detected.

| | >Thanks in advance, | | Good luck with your choices. Remember, you have to be compromised in the | first place for these exploits to take hold.

Not true. Spectre can be exploited via javascript on a web site (including an ad from who knows where).

Sorry forgot about the f*wit internet trampoline when I said that.

| In the weeks before this broke and while I was configuring Fedora 27 on | this new HW, I updated based on notices of important changes. | | No new updates since this news broke, so I did a refresh update. That

| produced no noticable changes. Mind you that's just ad hoc observations | from the system monitor and not benchmarked results. YMMV

There have been several relevant Fedora 27 updates. Some were released just as the first inklings were appearing.

In fact, the first news broke because an AMD patch to the kernel spilled the beans. The AMD patch turned off the already present mitigation for Meltdown in the case of AMD processors. --- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk

-- Russell

| From: D. Hugh Redelmeier via talk <talk@gtalug.org> | | | >And then there is the Spectre bug also lurking ... | | | | Microsoft has stopped, temporarily, updating AMD due to poor documentation. | | | | https://www.theverge.com/2018/1/9/16867068/microsoft-meltdown-spectre-security-updates-amd-pcs-issues> | | Thanks for this reference. I hadn't seen that. | | Microsoft isn't too clear about their finger-pointing (that doesn't | mean that they are wrong). This notice isn't very clear: | <https://support.microsoft.com/en-us/help/4073707/windows-os-security-update-block-for-some-amd-based-devices> | | - it would be really handy if they explained what "some" means. Which | AMD-based devices are currently blocked | | - this run-on sentence might not mean what it is supposed to | | After investigating, Microsoft has determined that some AMD | chipsets do not conform to the documentation previously | provided to Microsoft to develop the Windows operating system | mitigations to protect against the chipset vulnerabilities | known as Spectre and Meltdown. | | - Spectre and Meltdown are different. The fixes are different. | Preventing installation of the fixes for both is probably a mistake. | | To be honest, I don't understand why these mitigations involve a | chipset at all. The current best guess is that Microsoft forgot that early x86-64 machines did not have a CMPXCHG16B instruction. So patch KB4056897 used it and crashed. I base this on things referenced in the last paragraph of <https://arstechnica.com/gadgets/2018/01/bad-docs-and-blue-screens-make-microsoft-suspend-spectre-patch-for-amd-machines/> Alternatively, the problem MIGHT be that the processor has the instruction but that the chipset doesn't implement its part (lock?). I have read that Win8.1 and later require CMPXCHG16B so these problems are likely only showing up in Win 7 and Win 8 or perhaps even older systems. Is see the flag "cx16" in /proc/cpuid. I bet that tells me my CPU has the feature. <https://superuser.com/questions/187254/how-prevalent-are-old-x64-processors-lacking-the-cmpxchg16b-instruction>

On 01/11/2018 04:59 PM, Alvin Starr via talk wrote:

I cannot prove this yet but I believe there has been a big performance hit in the cloud providers.

I use amazon for a couple of lightly loaded servers

Attached is the traffic graph from the last 2 weeks.

Nothing has changed but for the reboot required by AWS as part of their patch roll out.

The reboot was at Tue Jan  2 09:02.

It may be just coincidence but the huge step-wise increase in base load is just crazy given nothing on my side changed but for the reboot.

So I take exception with Intel's comment that most folks will not notice the performance hit. Here is the graph.

-- Alvin Starr || land: (905)513-7688 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||

On 01/11/2018 05:29 PM, o1bigtenor wrote:

On Thu, Jan 11, 2018 at 3:59 PM, Alvin Starr via talk <talk@gtalug.org <mailto:talk@gtalug.org>> wrote:

I cannot prove this yet but I believe there has been a big performance hit in the cloud providers.

I use amazon for a couple of lightly loaded servers

Attached is the traffic graph from the last 2 weeks.

Nothing has changed but for the reboot required by AWS as part of their patch roll out.

The reboot was at Tue Jan  2 09:02.

It may be just coincidence but the huge step-wise increase in base load is just crazy given nothing on my side changed but for the reboot.

So I take exception with Intel's comment that most folks will not notice the performance hit.

Attached .png shows up here as a checkerboard.

Dee

ya. sorry about that. I posted a better version a bit later. -- Alvin Starr || land: (905)513-7688 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||

On 01/11/2018 06:21 PM, Jamon Camisso via talk wrote:

On 11/01/18 16:59, Alvin Starr via talk wrote:

...

I cannot prove this yet but I believe there has been a big performance hit in the cloud providers.

I use amazon for a couple of lightly loaded servers

Attached is the traffic graph from the last 2 weeks.

Nothing has changed but for the reboot required by AWS as part of their patch roll out.

The reboot was at Tue Jan  2 09:02.

It may be just coincidence but the huge step-wise increase in base load is just crazy given nothing on my side changed but for the reboot.

So I take exception with Intel's comment that most folks will not notice the performance hit. When you say server, is this a VM or bare metal? Is it dedicated or how is it provisioned on AWS?

The reason I ask is that if it is a VM, it could very well be the case that it was migrated to another host that may have higher resource contention. For example, what does steal show up as when you watch in top during the perceived slowdowns?

Its a paravirt VM. The steal time is much less than 1% bouncing between 0 and 0.2 Yes it is possible that other VM's are stealing resources leaving less compute power for my instance but I have had extremely stable cpu utilization going back something like 2 months. But the step is significant within a matter of a day it went from 2.5% utilization to 17.4% utilization in 2 steps. My image is held up by the moderator so I posted here ( https://owncloud.netvel.net/s/G3HGQUEb5saLBSn )if your interested. The timing is suspect and I had another server that sees periodic work and it completely collapsed because of the change in performance at just the same time. A bit of searching shows that others are complaining about the same problem. -- Alvin Starr || land: (905)513-7688 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||

On 01/11/2018 11:19 PM, Jamon Camisso via talk wrote:

On 11/01/18 19:13, Alvin Starr via talk wrote:

...

Its a paravirt VM. The steal time is much less than 1% bouncing between 0 and 0.2

Yes it is possible that other VM's are stealing resources leaving less compute power for my instance but I have had extremely stable cpu utilization going back something like 2 months. But the step is significant within a matter of a day it went from 2.5% utilization to 17.4% utilization in 2 steps.

My image is held up by the moderator so I posted here ( https://owncloud.netvel.net/s/G3HGQUEb5saLBSn )if your interested.

The timing is suspect and I had another server that sees periodic work and it completely collapsed because of the change in performance at just the same time.

A bit of searching shows that others are complaining about the same problem.

That does look like a substantial hit. What happens if you try the workload on a new VM? Roll the dice until you get lucky maybe?

This server is not so bad but it really shows the change in performance. The one that keeled over on me was on a small instance and I had to move it to a large instance. I did a bunch of database clean and hopefully that will be enough to be able to put it back on a small instance. Otherwise my costs at AWS till jump significantly. -- Alvin Starr || land: (905)513-7688 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||