Intel Optane DC persistent memory starts at $6.57 per GB

[Tabbykatze]

I work in enterprise network and infrastructure security so regularly am arm deep in their servers.

I'm not sure there will be that big a market for memory mode in general purpose because the latency is just not good enough for general use. It's too high for virtual machine deployment, the OS would be noticeably sluggish and have a poor user experience. It's also too high latency for financial applications on the stock exchange where every ms counts and increasing from average 30ns to 350ns would be drastic. I mean, i could see the memory mode dimms being used for paused vm offloading/swap replacement. And also for docker applications being loaded ad hoc but that would be more suitable in block storage.

Frankly, i can see definitive benefits in application and block storage mode, but what non latency intensive requirements are going to benefit a corporate environment. Research and dev are the primary areas i see the memory mode shining because of huge non latency sensitive datasets.

[DavidC1]

DRAM isn't at 30ns. You can get 70ns in PCs, and maybe 50ns if you super tweak them. Xeon Platinum servers with DDR gets 100ns.

The -EX chips with SMBs doubled that latency.

[MRFS]

> just mounting the entire OS C:/ drive onto that 128GB module would speed things up considerably

FYI: several years ago, we submitted a Provisional Patent Application for a "Format RAM" option to be added to a BIOS/UEFI subsystem. During a Windows fresh install, an upper-most region of memory addresses would be formatted as a ramdisk with C: drive letter, then Windows would install directly into that ramdisk in a transparent manner: properly implemented, the Windows install procedure would not know or care that the C: partition was located in DRAM. Now, with these Optane DIMMs, I should probably resurrect that Provisional Patent Application (it was way ahead of its time). Having what is in effect a non-volatile DIMM module is a perfect way to implement this "Format RAM" concept i.e. to host an OS in non-volatile memory that is much faster than NVMe SSDs.

[MRFS]

Here are 2 links to that Provisional Patent Application:
http://supremelaw.org/patents/bios.enhancements/provisional.application.1.htm
http://supremelaw.org/patents/bios.enhancements/provisional.application.2.htm
(I don't know if hexus.net blocks URLs to other websites: here goes ....)

[DanceswithUnix]

> just mounting the entire OS C:/ drive onto that 128GB module would speed things up considerably

FYI: several years ago, we submitted a Provisional Patent Application for a "Format RAM" option to be added to a BIOS/UEFI subsystem. During a Windows fresh install, an upper-most region of memory addresses would be formatted as a ramdisk with C: drive letter, then Windows would install directly into that ramdisk in a transparent manner: properly implemented, the Windows install procedure would not know or care that the C: partition was located in DRAM. Now, with these Optane DIMMs, I should probably resurrect that Provisional Patent Application (it was way ahead of its time). Having what is in effect a non-volatile DIMM module is a perfect way to implement this "Format RAM" concept i.e. to host an OS in non-volatile memory that is much faster than NVMe SSDs.

You seem to be heavily Windows leaning, so you might not be aware that in the Linux world it is very common for machines to boot from a ram drive. In embedded environments, the machine may continue running from the ramdrive until switched off. In server/desktop setups the ramdrive is kept small and once the machine has enough drivers loaded to complete the boot process the ramdrive is discarded.

I mention this for several reasons:
1/ Loading a ramdrive image into ram is time expensive, and during that time no actual booting (device setup) is getting done.
2/ The ram drive will contain things that are not in current use, wasting precious ram (hence initrd is usually discarded)
3/ Ram drives need to be closely tied to the operating system for good performance.

That last point needs expanding on. You have a binary on the ramdrive and you want to run it. If the OS has no idea what the ramdrive is, then it has to copy that binary into ram that it controls. It can demand page which helps, but there is a copy operation which costs time and potentially doubles the footprint of the binary. If the OS is in control of the ramdrive, then it can set an MMU memory page descriptor to the first 4K page of the binary, mark it as read only and executable, and start running. That gives trivial startup and page-in time and lower storage overheads.

So if you want to actually give your ideas a try, I suggest you go read up on initrd and tmpfs, here is an article you might want to start with: https://developer.ibm.com/articles/l-initrd/
I believe Vista caught up to some of this stuff, but Linux is just so much easier to hack about on with all the documentation and source code available. There is also a technique for which the name escapes me where you can boot from a read only filesystem like CD or DVD, and overlay a read/write layer on top to just pick up the changes. You will also find "bootchart" interesting, it shows how a modern OS pre-fetches configuration and binaries into RAM by instrumenting the previous boot so it knows what order those sectors were needed in. That pretty much gives you the fastest boot possible, the minimum fetching is done from SSD and things are in ram before they are actually required.


But back to Optane, it seems you can already partition part of the storage so I'm sure you could install and boot Linux off it. That really just makes it a faster SSD though, not exactly revolutionary.

Also, in data center use the working OS installs are virtualised with images in a network storage pool. That's the bit I'm finding hard to square with Optane. It could work well for Big Data applications, but big data doesn't seem a good fit for expensive storage.

Really I can only see Optane as a fast swap drive, allowing more idle/semi idle virtual machines in a VM farm box. I'm not convinced it is better than NVMe for that use though.

[Tabbykatze]

But back to Optane, it seems you can already partition part of the storage so I'm sure you could install and boot Linux off it. That really just makes it a faster SSD though, not exactly revolutionary.

Also, in data center use the working OS installs are virtualised with images in a network storage pool. That's the bit I'm finding hard to square with Optane. It could work well for Big Data applications, but big data doesn't seem a good fit for expensive storage.

Really I can only see Optane as a fast swap drive, allowing more idle/semi idle virtual machines in a VM farm box. I'm not convinced it is better than NVMe for that use though.

That's the problem I'm struggling to square my head around

[DanceswithUnix]

That's the problem I'm struggling to square my head around

Perhaps our hair just isn't pointy enough to embrace the paradigm synergies, or something

[Tabbykatze]

Perhaps our hair just isn't pointy enough to embrace the paradigm synergies, or something

The arcane magics of deep learning superservers are beyond the realms of normal mortal men!

[MRFS]

> 1/ Loading a ramdrive image into ram is time expensive, and during that time no actual booting (device setup) is getting done.

I'm not entirely sure why you have said that, unless you're
referring to a system that loads an OS image into RAM at startup;
in that case, I can see why the OS must be loaded entirely
before it's ready to do normal operating system tasks.

We've been using a 14GB ramdisk on our primary workstation
for several years now. The CPU is a quad-core Intel Q9550,
and the startup appears to be loading the ramdisk image
and doing other boot-up tasks in parallel.

But, that OS resides on a RAID-0 array of Samsung 2.5" SSDs,
and our Q9550 workstation is loading only a database into
the 14GB ramdisk in that system, NOT any working programs.

The software we are using is RamDisk Plus from www.superspeed.com .

I suspect that company has either gone out of business, or
their best technical staff have taken jobs with other IT companies.

Nevertheless, the RamDisk Plus driver runs as a single OS task
while the ramdisk image file is being loaded into DRAM.

One newer Windows 10 PC has 32GB of DDR3, and half of that is configured
as a 16GB ramdisk using SoftPerfect Ram Disk. That PC has been
working fine, after we finished adding some test software and
migrated that OS to a RAID-0 array with 4 x M.2 SSDs. The CPU
is a dual-core Intel Core i3 3300, with hyper-threading,
which we intend to upgrade later, after our testing is done.

The latter PC also loads the ramdisk image file at startup
in parallel with other boot-up tasks (same database as the
one on the Q9550 workstation).


Many thanks for your feedback.


p.s. Please understand that our original Provisional Patent
Applications were dated 2007 and 2008 (11-12 years ago).

[MRFS]

Please allow me to configure an Optane Workstation, using the
following feasible assumptions:

(1) start with a standard ATX motherboard form factor

(2) add 3 DIMM slots to the left of the CPU socket, and
add 3 DIMM slots to the right of the CPU socket -- 6 DIMM slots total

(3) the pairs of DIMM slots closest to the CPU host standard DDR4 DIMMs
-- 4 DIMM slots total in quad-channel mode

(4) the left-most and right-most DIMM slots host Optane DIMMs
-- 2 DIMM slots total in dual-channel mode

(5) Intel begins to manufacture a 16GB Optane DIMM, a 32GB Optane DIMM, and a 64GB Optane DIMM
intended for the prosumer and workstation market niche

(6) 4 x DDR4 @ 16GB = 64GB of modern DDR4 DRAM

(7) 2 x Optane @ 32GB = 64GB of Optane DIMMs

(8) assemble all hardware, Power On Self-Test, and boot into the UEFI MB BIOS;

(9) choose the "Format RAM" option, and create an NTFS C: partition on the Optane 64GB

(10) after formatting is completed, install Windows directly to that C: partition

(11) shutdown and startup again, to test that all settings are AOK

Comments:

some ASUS motherboards already host 2 x DIMM.2 slots

Intel should have no trouble populating their Optane DIMMS with only 16GB, 32GB and 64GB;
those three capacities should reduce their retail prices a LOT

the UEFI BIOS should be smart enough to allow the builder to format 2 x NTFS partitions,
one for C:, and one for a back-up copy of the OS if/when C: fails e.g. with malware

choosing one or the other OS is a simple BIOS setting, just like now

for example,
2 x OS partitions @ 32GB = 64GB of Optane DIMMs
2 x OS partitions @ 64GB = 128GB of Optane DIMMs

Intel retains proprietary rights to their Optane DIMMs,
to compete more effectively with AMD in the workstation market


All of the above are on "this side" of the visible horizon
even if they do not become a reality for another 12 to 24 months.

Yes, all of the above assume that Windows is the intended OS.

[MRFS]

MB w/ 2 x DIMM.2 slots:

ASUS ROG Dominus Extreme Intel LGA 3647 for Xeon W-3175X (C621) 12 DIMM DDR4 DIMM.2 U.2 EEB Performance Motherboard with Aquantia 10G LAN, USB 3.1

https://media3.webcollage.net/3e4b5036224a71ae9f0cc1e02c32a1e3ff37cfcc?response-content-type=image%2Fpng&AWSAccessKeyId=AKIAIIE5CHZ4PRWSLYKQ&Expires=1893486652&Signature=S1NBYFkxPMUkRRk6cV CVUtAjpa4%3D

https://www.newegg.com/Product/Product.aspx?Item=N82E16813119192&Tpk=N82E16813119192

[DanceswithUnix]

But, that OS resides on a RAID-0 array of Samsung 2.5" SSDs,
and our Q9550 workstation is loading only a database into
the 14GB ramdisk in that system, NOT any working programs.

I'm puzzled, usually a database program keeps any working set of the data in ram anyway. So it sounds like you have a copy of the database in ramdisk, as well as the copy that the database program should hold in ram. Unless you have an ancient and very bad database, in which case I would recommend PostgreSQL as fast and free (and comes packaged as part of most Linux distributions for easy update on your database server). Then unless your database is read-only, you have the problem of backing up the ramdisk contents back to SSD in a transaction safe manner. All sounds rather fraught.

What happens if you run the database direct from SSD without the ramdrive?

This isn't the 1980's, for most tasks the ramdisk is dead as it ties up ram for one purpose and so slows the machine down compared with letting the operating system cache the data and look after ram use itself. Where ramdisks are used, they should be dynamically scaled so they use the minimum ram possible for storing what is on there rather than having a pre-configured amount of space. That fits with the idea that you should only store things you don't care losing should the power fail. Google tmpfs for more.

p.s. Please understand that our original Provisional Patent
Applications were dated 2007 and 2008 (11-12 years ago).

Please understand that the ram booting tools I mentioned above were well established and mainstream by then.

[MRFS]

> What happens if you run the database direct from SSD without the ramdrive?

Ours is not a structured "database program" like an SQL database:

it's a set of about 150,000 discrete files of all different types,
about half of which are hyper-linked HTML files + many other
Windows filename "extents" such as .doc, .txt, .gif, .jpg, etc.

In general we try to do as much routine work with the ramdisk,
because it minimizes wear that would otherwise occur with our SSDs.

For example, we use COPERNIC to locate files with certain contents,
and the COPERNIC indexing task really flies when reading the ramdisk.

In answer to your question above, the set of files on the RAID-0 array of SSDs
more or less mirrors the contents of the ramdisk. To guarantee that they
are "synced" we simply run a short batch program that launches XCOPY:

R:
xcopy R:\folder E:\folder /s/e/v/d
E:
xcopy E:\folder R:\folder /s/e/v/d

That short batch file uses command line tokens
if we need to "sync" files across our LAN
which functions mainly as backup file servers.

We prefer the ramdisk because simple tasks like browsing that many files
goes MUCH faster with the ramdisk.

And, this speed is even faster with our Windows 10 PC that now has
32GB of DDR3, half of which is a SoftPerfect Ram Disk. That Windows 10
PC is a design we intend to replicate with an AMD Threadripper system
e.g. ASRock X399M with the ASRock Ultra Quad M.2 "4x4" add-in card.

For example, let's say I'm looking for filenames that contain some text string.
From experience, we've found that the Windows ATTRIB program can locate
matching filenames much more quickly than any other method we have tried
without writing custom software e.g.

attrib *2018-04-14.gif" /s

... is almost instantaneous with results when executed on the ramdisk,
as opposed to any other storage devices.


> Please understand that the ram booting tools I mentioned above were well established and mainstream by then.

Thanks for that: I've been using MS Windows ever since the first release of MS-DOS and
I have no experience whatsoever with Linux or with configuring servers.

I don't believe those "ram booting tools" were available for Windows end users, however:
please correct me if I am wrong about that. Over the years, I have tried to do a
lot of reading, to stay current with IT news; as such, I probably would have noticed
if such tools had become commercially available for Windows XP et seq.


We would like to experiment with the "Optane Workstation"
which I visualized above. Expensive new electronic technologies
usually fall in price with time and volume manufacturing.

So, I'm willing to wait for smaller capacity Optane DIMMs
to become available for Windows workstations.

Because I started using computers in 1971, I make a habit
of telling IT vendors what I would like to have:
sometimes they do it, sometimes not.

On this point, I would like to see Optane become available on the SO-DIMM form factor
because our utility patent was designed in anticipation of non-volatile RAM.

Nothing ventured / nothing gained

Thanks again!

[MRFS]

One more comment, if you'll bear with me:

if Intel does NOT consent to making Optane DIMMs
with lower capacities e.g. 16GB, 32GB and 64GB,
that decision will speak volumes about
Intel's overall market focus on data centers.

Because they have already announced 128GB and 256GB Optane DIMMs,
it should be a piece o' cake to manufacture the smaller capacities.

However, if Optane is prevented from working in AMD systems,
-- for proprietary / intellectual property reasons --
I guess I'll just have to "make do" with non-volatile RAM
developed by some other manufacturer, like Samsung.

FYI: here's the utility patent that was finally awarded
after active obstruction by corrupt government attorneys:

http://supremelaw.org/patents/BayRAMFive/utility.application/award/

Next evolution of that patent is planned to replace SATA cables
with PCIe 4.0 U.2 cables @ 16GHz, and function as well with non-volatile SO-DIMMs
that do not need a secondary PSU to retain data after shutdown.

[MRFS]

I do prefer to look towards the future,
going forward, particularly with a focus
on the practical potential of non-volatile
DRAM in computer workstations.

My favorite bumper sticker reads:

CONSTANT CHANGE IS HERE TO STAY!

A DDR4-4000 DRAM bus is capable of 4,000 x 8 = 32,000 MB/second.

At PCIe 4.0, x16 lanes @ 16G / 8.125 bits per byte = 31,500 MB/second.

As such, hosting a workstation OS on a "4x4" ASRock Ultra Quad M.2
add-in card should function almost exactly the same as an
OS that is stored in a non-volatile ramdisk, using Optane DIMMs.

At YouTube, der8auer demonstrated 8 x Samsung NVMe M.2 SSDs
using 2 x ASUS 4x4 cards. The doubled clock rate of PCIe 4.0
should make the same possible with only 1 x ASUS 4x4 card.

That solution will make an Optane Workstation with OS-in-ramdisk
unnecessary, because the same performance can be achieved
with less expensive parts from other vendors.

[MRFS]

So, it may be quite a while before Optane DIMMs
can work in mid-range workstations, if ever:

"It should be noted that not all of Intel's second generation Xeon Scalable CPUs support Optane. Only Xeon Platinum 8200 family, Xeon Gold 6200 family, Xeon Gold 5200 family, and the Xeon Silver 4215 does. The Xeon Platinum 9200 family do not."

Source: https://www.anandtech.com/show/14155/intels-enterprise-extravaganza-2019-roundup

... hence my preference to build a TR workstation
with the ASRock Ultra Quad M.2 add-in card
hosting the OS.