Booting Fedora from M.2 Optane Nvme
With all the CPU cache fencing going on I thought it might be fun to fiddle with NVM express on Fedora 27. My primary SSD on this z370-A, was/is in build limbo while I updated bios and waited for outcomes. I installed a 32gib module from Newegg ($90 delivered) and I booted from F27-workstation live on usb. I let the anaconda installer do the automatic partitioning. 5gib of the 32gib reported was left unallocated for over-provisioning. First boot appeared to be twice as fast as on the SATA SSD. Three or four seconds to get to the gui login as opposed to 7 or 8 on the original drive. Had to install hdparm from the repo. # hdparm -tT --direct /dev/sda1 /dev/sda1: Timing O_DIRECT cached reads: 930 MB in 2.00 seconds = 464.92 MB/sec Timing O_DIRECT disk reads: 1370 MB in 3.00 seconds = 456.30 MB/sec # hdparm -tT --direct /dev/nvme0n1 /dev/nvme0n1: Timing O_DIRECT cached reads: 2302 MB in 2.00 seconds = 1151.84 MB/sec Timing O_DIRECT disk reads: 3474 MB in 3.00 seconds = 1157.65 MB/sec More info on Nvme with a nice diagram of the Linux storage stack here. https://en.m.wikipedia.org/wiki/NVM_Express I switched back to the original SSD. An os-probe mapped the device to /dev/dm-1 and I was able to remake grub.conf and add this install to the bootloader on that drive. A couple of switches back and forth between drives and it looks like the 10% performance hit I had experienced after the microcode updates, is now somewhat reduced, +-5%. Guilding the lilly a bit, as I didn't have any day to day problems with performance. Although it's hard to name Intel's product actions around this issue a field of Lilly's. -- Russell
On Sun, Mar 18, 2018 at 12:32:07PM -0400, Russell via talk wrote:
With all the CPU cache fencing going on I thought it might be fun to fiddle with NVM express on Fedora 27. My primary SSD on this z370-A, was/is in build limbo while I updated bios and waited for outcomes.
I installed a 32gib module from Newegg ($90 delivered) and I booted from F27-workstation live on usb. I let the anaconda installer do the automatic partitioning. 5gib of the 32gib reported was left unallocated for over-provisioning.
First boot appeared to be twice as fast as on the SATA SSD. Three or four seconds to get to the gui login as opposed to 7 or 8 on the original drive.
Had to install hdparm from the repo.
# hdparm -tT --direct /dev/sda1
/dev/sda1: Timing O_DIRECT cached reads: 930 MB in 2.00 seconds = 464.92 MB/sec Timing O_DIRECT disk reads: 1370 MB in 3.00 seconds = 456.30 MB/sec
# hdparm -tT --direct /dev/nvme0n1
/dev/nvme0n1: Timing O_DIRECT cached reads: 2302 MB in 2.00 seconds = 1151.84 MB/sec Timing O_DIRECT disk reads: 3474 MB in 3.00 seconds = 1157.65 MB/sec
More info on Nvme with a nice diagram of the Linux storage stack here.
https://en.m.wikipedia.org/wiki/NVM_Express
I switched back to the original SSD. An os-probe mapped the device to /dev/dm-1 and I was able to remake grub.conf and add this install to the bootloader on that drive.
A couple of switches back and forth between drives and it looks like the 10% performance hit I had experienced after the microcode updates, is now somewhat reduced, +-5%.
Guilding the lilly a bit, as I didn't have any day to day problems with performance. Although it's hard to name Intel's product actions around this issue a field of Lilly's.
Not bad, but I think for my money I would get an EVO 960 instead. Twice the speed and 8 times the space for just under twice the money. Or 16 times the space for three times the money. I just don't see any advantage to the Optane at this point. -- Len Sorensen
On March 19, 2018 9:32:01 AM EDT, lsorense@csclub.uwaterloo.ca wrote:
With all the CPU cache fencing going on I thought it might be fun to fiddle with NVM express on Fedora 27. My primary SSD on this z370-A, was/is in build limbo while I updated bios and waited for outcomes.
I installed a 32gib module from Newegg ($90 delivered) and I booted from F27-workstation live on usb. I let the anaconda installer do the automatic partitioning. 5gib of the 32gib reported was left unallocated for over-provisioning.
First boot appeared to be twice as fast as on the SATA SSD. Three or four seconds to get to the gui login as opposed to 7 or 8 on the original drive.
Had to install hdparm from the repo.
# hdparm -tT --direct /dev/sda1
/dev/sda1: Timing O_DIRECT cached reads: 930 MB in 2.00 seconds = 464.92 MB/sec Timing O_DIRECT disk reads: 1370 MB in 3.00 seconds = 456.30 MB/sec
# hdparm -tT --direct /dev/nvme0n1
/dev/nvme0n1: Timing O_DIRECT cached reads: 2302 MB in 2.00 seconds = 1151.84 MB/sec Timing O_DIRECT disk reads: 3474 MB in 3.00 seconds = 1157.65 MB/sec
More info on Nvme with a nice diagram of the Linux storage stack here.
https://en.m.wikipedia.org/wiki/NVM_Express
I switched back to the original SSD. An os-probe mapped the device to /dev/dm-1 and I was able to remake grub.conf and add this install to
On Sun, Mar 18, 2018 at 12:32:07PM -0400, Russell via talk wrote: the bootloader on that drive.
A couple of switches back and forth between drives and it looks like
the 10% performance hit I had experienced after the microcode updates, is now somewhat reduced, +-5%.
Guilding the lilly a bit, as I didn't have any day to day problems
with performance. Although it's hard to name Intel's product actions around this issue a field of Lilly's.
Not bad, but I think for my money I would get an EVO 960 instead. Twice the speed and 8 times the space for just under twice the money. Or 16 times the space for three times the money. I just don't see any advantage to the Optane at this point.
Yea, this is just a tinker-toy. I had figured to put a small backup OS on this, remove its allocation from the chainloader and then leave it alone in the box in case I need it, instead of using a live usb for troubleshooting etc. It doesn't look like I can mask it's physical presence tho, except by populating a pcie slot and co-opting it's assigned pci lanes, so no security through obscurity. Instead I went with full Fedora w/gnome. I have 7gib on the drive after a full workstation install so I may rethink stuff and use it as my primary. Optane was meant to be a cache accelerator for booting Windows 10 on a standard drive and thats about the sum of it. Although, with all the elbow room and thumbscrews in this box, you could use it as a key drive for sneakernet backup to a safety deposit box pretty handily. You'd just need the right finger screw for the 1_M.2 slot standoff post, which is exposed. The other slot is buried under a heat sink.
-- Len Sorensen
-- Russell
On Mon, Mar 19, 2018 at 11:02:06AM -0400, Russell wrote:
Yea, this is just a tinker-toy. I had figured to put a small backup OS on this, remove its allocation from the chainloader and then leave it alone in the box in case I need it, instead of using a live usb for troubleshooting etc.
It doesn't look like I can mask it's physical presence tho, except by populating a pcie slot and co-opting it's assigned pci lanes, so no security through obscurity.
Instead I went with full Fedora w/gnome. I have 7gib on the drive after a full workstation install so I may rethink stuff and use it as my primary. Optane was meant to be a cache accelerator for booting Windows 10 on a standard drive and thats about the sum of it.
Although, with all the elbow room and thumbscrews in this box, you could use it as a key drive for sneakernet backup to a safety deposit box pretty handily. You'd just need the right finger screw for the 1_M.2 slot standoff post, which is exposed. The other slot is buried under a heat sink.
Well as a toy, $90 is easier to justify than $170. Not sure how many insertions that M.2 slot is rated for. I don't think I would do anything that involves frequent swapping. An external thunderbolt connected NVMe drive would survive that much better. -- Len Sorensen
On March 19, 2018 11:20:09 AM EDT, lsorense@csclub.uwaterloo.ca wrote:
Yea, this is just a tinker-toy. I had figured to put a small backup OS on this, remove its allocation from the chainloader and then leave it alone in the box in case I need it, instead of using a live usb for
On Mon, Mar 19, 2018 at 11:02:06AM -0400, Russell wrote: troubleshooting etc.
It doesn't look like I can mask it's physical presence tho, except by
populating a pcie slot and co-opting it's assigned pci lanes, so no security through obscurity.
Instead I went with full Fedora w/gnome. I have 7gib on the drive
after a full workstation install so I may rethink stuff and use it as my primary. Optane was meant to be a cache accelerator for booting Windows 10 on a standard drive and thats about the sum of it.
Although, with all the elbow room and thumbscrews in this box, you
could use it as a key drive for sneakernet backup to a safety deposit box pretty handily. You'd just need the right finger screw for the 1_M.2 slot standoff post, which is exposed. The other slot is buried under a heat sink.
Well as a toy, $90 is easier to justify than $170.
Not sure how many insertions that M.2 slot is rated for. I don't think I would do anything that involves frequent swapping.
I wouldn't recommend it commercially or for production, but personal privacy may take a lot of different forms and methods. In a sense nonstandard can be a good thing. There is a pcie adapter for M.2 so I imagine that some amount of removal and insertion is built into the lifespan of the product, albeit without warranty of purpose.
An external thunderbolt connected NVMe drive would survive that much better.
In fact I just wanted to look at a measure of cache performance mitigations for now. Onboard recovery OS or, deposit box storage of the card w/os & data, are just a way of self justifing the cost of playing around with Linux and perhaps extending the scope of my tinkering. Externals are possibly subject to theft, loss and impact damage from falling off the desk. Stuff attached on the inside of the box, is just a bit more inconvienent to access and that makes it a little more physically secure. Three second delay for bios and three more seconds to login prompt, pretty much changed my mind about useage of this card. I'll probably use it as my primary drive for the time being. There is an apparent improvement in browsing the web. Seems like pages load faster than on the SSD, but that may just be wishfull thinking. All in good fun.
-- Len Sorensen
-- Russell
On Mon, Mar 19, 2018 at 01:55:56PM -0400, Russell wrote:
I wouldn't recommend it commercially or for production, but personal privacy may take a lot of different forms and methods. In a sense nonstandard can be a good thing.
There is a pcie adapter for M.2 so I imagine that some amount of removal and insertion is built into the lifespan of the product, albeit without warranty of purpose.
I would not be surprised if the M.2 connector has a rating of 50 insertions. It is obviously designed for internal use and not frequent changes.
In fact I just wanted to look at a measure of cache performance mitigations for now.
Onboard recovery OS or, deposit box storage of the card w/os & data, are just a way of self justifing the cost of playing around with Linux and perhaps extending the scope of my tinkering.
Externals are possibly subject to theft, loss and impact damage from falling off the desk. Stuff attached on the inside of the box, is just a bit more inconvienent to access and that makes it a little more physically secure.
Three second delay for bios and three more seconds to login prompt, pretty much changed my mind about useage of this card.
If only the BIOS could be faster.
I'll probably use it as my primary drive for the time being. There is an apparent improvement in browsing the web. Seems like pages load faster than on the SSD, but that may just be wishfull thinking.
All in good fun.
Well some browsers do like to maintain their state in databases in your home directory so a fast disk sure could help. -- Len Sorensen
On March 19, 2018 2:23:31 PM EDT, lsorense@csclub.uwaterloo.ca wrote:
I wouldn't recommend it commercially or for production, but personal
On Mon, Mar 19, 2018 at 01:55:56PM -0400, Russell wrote: privacy may take a lot of different forms and methods. In a sense nonstandard can be a good thing.
There is a pcie adapter for M.2 so I imagine that some amount of
removal and insertion is built into the lifespan of the product, albeit without warranty of purpose.
I would not be surprised if the M.2 connector has a rating of 50 insertions. It is obviously designed for internal use and not frequent changes.
Probably the same with the pcie cards, but if the data backups for offsite (personal data) are once or twice a year, with carefull handling the card could probably last a generation.
In fact I just wanted to look at a measure of cache performance mitigations for now.
Onboard recovery OS or, deposit box storage of the card w/os & data, are just a way of self justifing the cost of playing around with Linux and perhaps extending the scope of my tinkering.
Externals are possibly subject to theft, loss and impact damage from falling off the desk. Stuff attached on the inside of the box, is just a bit more inconvienent to access and that makes it a little more physically secure.
Three second delay for bios and three more seconds to login prompt, pretty much changed my mind about useage of this card.
If only the BIOS could be faster.
Actually the three seconds is the delay to allow the user to hit F2 to enter bios. Could probably reduce that to one, I'll have to check that.
I'll probably use it as my primary drive for the time being. There is an apparent improvement in browsing the web. Seems like pages load faster than on the SSD, but that may just be wishfull thinking.
All in good fun.
Well some browsers do like to maintain their state in databases in your home directory so a fast disk sure could help.
Sure seems to. I had intended to put in a larger SATA platter and move my old data onto it. I think now I'll wait a bit longer and see where the CAD vs. tech price point ends up over the next little while and go directly to nvme. That is hoping the dollar doesn't tank any further (unlikely), the loonie hasn't been this low in quite a while and also that the bitcoiners stop buying out the stock (also unlikely) but we'll see.
-- Len Sorensen
-- Russell
participants (2)
-
lsorense＠csclub.uwaterloo.ca -
Russell