Overview and usage of 「EthFS: Ethereum File System」, which expands the range of fully on-chain NFTs.

Hello, I’m fully on-chain NFT creator nawoo.

This article describes EthFS: Ethereum File System, which allows you to easily store files in the Ethereum chain.

nawoo

With EthFS, you can do the following

• Easily upload files from the official website
  • Saves gas costs compared to storing in storage because it uses SSTORE2
  • Any type of file can be uploaded, be it an image, text file, JavaScript, or anything else!
  • There is no upload size limit, and files as large as almost 300KB have been uploaded!
• Uploaded files can be freely used by anyone
  • You can check the list of files and their details in the File Explorer on the official site.
  • You can easily load files from the contract

The creator is frolic, who is also a member of the MUD development team.

あわせて読みたい

自律型世界（オンチェーンゲーム）の構築をサポートする「MUD」の概要と、MUDを用いた主要ゲームプロジ… 昨今では、世界中のクリプトコアな人たちを中心に「オンチェーンゲーム開発」がおこなわれており、弊メディアでもDark ForestやIsaacなど先進的な事例についてピックア…

Libraries such as Three.js and p5.js have already been uploaded and can be used to create fully on-chain NFTs such as ROSES.

This article explains how to use EthFS, the contents of the two contracts, ContentStore and FileStore, which are the main body of EthFS, and how to actually create a fully on-chain NFT using EthFS.

Let us begin by describing the structure of this article.

I hope that this article will be of some help to those who are interested in understanding the Overview of EthFS, its usage, and the possibility of building a fully on-chain NFT.

How to use EthFS

Upload Files

You can easily upload files from the official site.

• Mainnet version: ethfs.xyz
• Testnet version: goerli.ethfs.xyz

nawoo

This time, I actually tried uploading files using the test-net version.

First, after Connect Wallet, click the button in the center of the File Uploader window.

Then select the file you wish to upload. In this case, we will use the image file “earth.jpg”.

As a reminder, files are now always Base64 encoded when uploaded from the official site.

If you want to upload files without Base64 encoding, you must handle contracts directly.

nawoo

I will explain how to do this later.

According to frolic, a checkbox will be added to the upload screen in the near future that will allow the user to choose whether to Base64 encode or not.

You will see that the image file “earth.jpg” is 61KB in size, but 81KB when Base64 encoded. Fill in the file’s license information, check the agreement, and then click the Upload button.

MetaMask will then start and you will click Confirm several times. In this example, a total of five transactions will be issued to split an 81KB file into four separate uploads and then combine them into a file at the end.

nawoo

As we will explain later, this means that this is a transaction to execute the addContent function of the ContentStore contract four times and the createFile function of the FileStore contract.

Wait for a while, and when “FileCreated!” appears in the lower right corner, you are done.

earth.jpg has been added to the File Explore window.

Double-click on the “earth.jpg” file in the File Explorer window to view the image and more information.

About bounty

In the lower right corner of the official website, there is an icon called File Bounties. Clicking on it brings up the following screen.

p5.js and Three.js were uploaded to the mainnet through a “bounty” mechanism.

This mechanism is intended to spread the cost of gas by splitting large files and having many people upload them.

nawoo

I also participated in the Three.js (r147) bounty and uploaded one. There is no specific reward, but my contribution was recorded as a transaction.

We expect the bounty to continue in the future, so interested parties are encouraged to follow @frolic.

Loading Files

To use an uploaded file, use the FileStore contract. getFile(filename).read() with the file name.

For example, to create a fully on-chain NFT using Three.js, the code to load Three.js would look like this

// Example of loading Three.js (r147)
fileStore.getFile("three-v0.147.0.min.js.gz").read()

あわせて読みたい

Three.jsを使った3DフルオンチェーンNFTを作ってみよう【開発】 本記事では、以前取り上げたdom氏の制作した3DフルオンチェーンNFT「ROSES」のような、Three.jsを使ったフルオンチェーンNFTを作成する方法について解説します。 一般的…

This Three.js is provided gzip compressed and Base64 encoded. Therefore, to use Three.js, you need the code to restore it. However, that code has already been uploaded to EthFS.

nawoo

For more information, let’s look at the following ROSES example.

Example of ROSES

あわせて読みたい

フルオンチェーンNFTの新時代を切り拓くか｜dom氏が作成した「ROSES」について徹底解説 先日、dom氏(@dhof)が Twitter上で、新作NFTとなる「ROSES」を発表しました。 ROSESprocedurally generated in 3d + served fully on-chainsupply 1024 (32 reserved)NO…

The EthFS repository (GitHub) has a publicly available example of ROSES reproduction using EthFS.

Compared to the original ROSES code, you can see that it is much simpler. It just reads the three files from EthFS and combines them with string.concat.

• three.min.js.gz → gzip compressed Three.js (uploaded to testnet)
• gunzipScripts → JS code for gzip expansion
• rose.js → main code for ROSES

contract RoseExample is ERC721 {
    IFileStore public immutable fileStore;

    // ＊snip＊

    function tokenURI(uint256 tokenId)
        public
        view
        virtual
        override
        returns (string memory)
    {
        return string.concat(
            "data:application/json,%7B%22name%22%3A%22Example%20Rose%22%2C%22animation_url%22%3A%22data%3Atext%2Fhtml%2C%253Cscript%250A%2520%2520type%253D%2522text%252Fjavascript%252Bgzip%2522%250A%2520%2520src%253D%2522data%253Atext%252Fjavascript%253Bbase64%252C",
            fileStore.getFile("three.min.js.gz").read(), // Load three.js from EthFS
            "%2522%250A%253E%253C%252Fscript%253E%250A%253Cscript%2520src%253D%2522data%253Atext%252Fjavascript%253Bbase64%252C",
            fileStore.getFile("gunzipScripts.js").read(), // Load JavaScript for gzip decompression from EthFS
            "%2522%253E%253C%252Fscript%253E%250A%250A%253Cscript%253E%250A%2520%2520var%2520tokenId%2520%253D%2520",
            toString(tokenId),
            "%253B%250A%253C%252Fscript%253E%250A%253Cstyle%253E%250A%2520%2520*%2520%257B%250A%2520%2520%2520%2520margin%253A%25200%253B%250A%2520%2520%2520%2520padding%253A%25200%253B%250A%2520%2520%257D%250A%2520%2520canvas%2520%257B%250A%2520%2520%2520%2520width%253A%2520100%2525%253B%250A%2520%2520%2520%2520height%253A%2520100%2525%253B%250A%2520%2520%257D%250A%253C%252Fstyle%253E%250A%253Cscript%2520src%253D%2522data%253Atext%252Fjavascript%253Bbase64%252C",
            fileStore.getFile("rose.js").read(), // Load ROSES main code from EthFS
            "%2522%253E%253C%252Fscript%253E%250A%22%7D"
        );
    }
}

The string.concat section is URL encoded and difficult to read, so here is the decoded version.

'data:application/json,{"name":"Example Rose","animation_url":"data:text/html,<script type="text/javascript+gzip" src="data:text/javascript;base64,"',
fileStore.getFile("three.min.js.gz").read(), // Load three.js from EthFS
'"></script><script src="data:text/javascript;base64,',
fileStore.getFile("gunzipScripts.js").read(), // Load JavaScript for gzip decompression from EthFS
'"></script><script>var tokenId = ',
toString(tokenId),
';</script><style>* { margin: 0; padding: 0; } canvas { width: 100%; height: 100%; }</style><script src="data:text/javascript;base64,',
fileStore.getFile("rose.js").read(), // Load ROSES main code from EthFS
'"></script>"}'

Script tags with type="text/javascript+gzip" will be gzipped by gunzipScripts.js.

Explanation of Contracts

The main body of EthFS is two contracts, ContentStore and FileStore.

EthFS uses SSTORE2 (data contract), but since data contracts have a size limit of 24KB, files are split into smaller than 24KB before uploading.

ContentStore is used to upload each divided part, and FileStore is used to group them together as a single file.

About SSTORE2

nawoo

SSTORE2 was introduced in the following article, but a brief review is in order.

あわせて読みたい

不可能を可能にする開発テクニック｜724KBのJPG画像をフルオンチェーンNFTにする方法と具体的な手順を解… 先日、ニュージーランド在住のフォトグラファーShinichi MaruyamaさんのNFT作品 “Light Sculpture” が公開されました。 「Light Sculpture」は永続性をコンセ…

• Saves gas compared to storing to storage because it is stored as a data contract
• Maximum size is 24,575 bytes
• Data is saved using the write function, which returns the address (pointer) of the data contract as the return value
• Use the read function to read data

Upload Files

nawoo

Let’s look at how the two contracts are used when uploading a file.

The maximum size of data that can be stored in SSTORE2 is 24,575 bytes; if you wish to upload a file larger than that, you must split the file in advance.

File splitting is done on the local PC or front-end side, not on the contract.

addContent function

Upload each divided file part (chunk) with the addContent function of the ContentStore contract.

The addContent function receives an argument (content) of type bytes and calculates a hash value with the keccak256 function as a checksum.

nawoo

If the checksum is already registered, exit without uploading; if it is not registered, save it in the data contract with SSTORE2.

The return value of SSTORE2.write is the address of the data contract (pointer), so the checksum and the SSTORE2 pointer are stored in mapping.

mapping(bytes32 => address) public pointers;　// checksum => Mapping of SSTORE2 pointer

function addContent(bytes memory content) public returns (bytes32 checksum, address pointer) {
    checksum = keccak256(content); // Find the hash value and use it as a checksum
    if (pointers[checksum] != address(0)) {
        return (checksum, pointers[checksum]); // If the hash value has already been registered, do not upload it.
    }
    pointer = SSTORE2.write(content); // Save data with SSTORE2
    pointers[checksum] = pointer; // Store checksum and SSTORE2 pointer in mapping
    emit NewChecksum(checksum, content.length);
    return (checksum, pointer);
}

Save the parts (chunks) of all files with addContent and record the checksum of the return value. This checksum is used in the next step.

createFile function

Next, file information is created with the createFile function of the FileStore contract.

nawoo

The arguments are the file name, an array of file part checksums, and file metadata (extraData).

mapping(string => bytes32) public files; // File name => checksum mapping of File structure

function createFile(
    string memory filename, // File name
    bytes32[] memory checksums, // Checksum of each chunk (uploaded to ContentStore)
    bytes memory extraData // File metadata (media type and license)
) public returns (File memory file) {
    if (files[filename] != bytes32(0)) { // Check if the file name is not already registered
        revert FilenameExists(filename);
    }
    return _createFile(filename, checksums, extraData);
}

• createFile function:
  • Call _createFile function to make sure the file name is not already registered.
• _createFile function:
  • Calculates the size of the file and creates a File structure (struct)

function _createFile(
    string memory filename,
    bytes32[] memory checksums,
    bytes memory extraData
) private returns (File memory file) {
    Content[] memory contents = new Content[](checksums.length);
    // Calculate file size
    uint256 size = 0;
    for (uint256 i = 0; i < checksums.length; ++i) {
        size += contentStore.contentLength(checksums[i]);
        contents[i] = Content({checksum: checksums[i], pointer: contentStore.getPointer(checksums[i])});
    }
    if (size == 0) {
        revert EmptyFile();
    }
    // Create a File structure and register it in ContentStore
    file = File({size: size, contents: contents});
    (bytes32 checksum, ) = contentStore.addContent(abi.encode(file));
    // Save file name and checksum of File structure to mapping
    files[filename] = checksum;
    // Issue FileCreated event
    emit FileCreated(filename, checksum, filename, file.size, extraData);
}

The File structure (struct) is defined as follows:

struct Content {
    bytes32 checksum; // Chunk checksum
    address pointer; // Address of data contract
}
struct File {
    uint256 size; // File size
    Content[] contents; // Array of Content
}

After uploading the File structure again with ContentStore‘s addContent function, the file name and the checksum of the File structure are stored in mapping.

The file metadata (extraData) passed as the third argument of createFile is in JSON format and includes media type and license information.

nawoo

For example, there is.

// 例: Metadata of earth.jpg
{
   "type": "image/jpeg",
   "encoding": "base64",
   "license": "CC0"
}

Interestingly, this extraData is not stored in the data contract or in storage. It is only used when the FileCreated event is issued.

Therefore, if metadata for a file is to be retrieved at the front end or elsewhere, it must be retrieved from the event log. The decision may be made that recording less important data in the event log is sufficient to save gas.

Loading Files

To read a file from a contract, you must use the getFile and read functions and specify the file name.

fileStore.getFile("three.min.js.gz").read()

getFile Function

Let’s take a look at the getFile function of the FileStore contract.

nawoo

The return value of the function is of type File. The checksum is obtained from the file name and the File structure is read from the checksum.

function getFile(string memory filename) public view returns (File memory file) {
    bytes32 checksum = files[filename]; // Get checksum from file name
    if (checksum == bytes32(0)) {
        revert FileNotFound(filename); // Error if file name is unregistered
    }
    address pointer = contentStore.pointers(checksum); // Get SSTORE2 pointer from checksum
    if (pointer == address(0)) {
        revert FileNotFound(filename); // Error if checksum is unregistered
    }
    return abi.decode(SSTORE2.read(pointer), (File)); // Read File structure from SSTORE2
}

read function

The read function of the File structure is defined in File.sol.

Each part of a file saved using SSTORE2 is read and combined. At this time, we do not use the SSTORE2.read function, but use Inline Assembly to write directly to the memory area.

function read(File memory file) view returns (string memory contents) {
    // Get an array of each part (chunk) of the file
    Content[] memory chunks = file.contents;
    assembly {
        let len := mload(chunks) // Get the length of the array stored in the header
        let totalSize := 0x20 // Add the first 32 bytes of the header
        contents := mload(0x40) // Read from free memory pointer
        let size
        let chunk
        let pointer

        // Loop as many times as the number of chunks
        for { let i := 0 } lt(i, len) { i := add(i, 1) } {
            // Obtain the i-th chunk (=Content structure)
            chunk := mload(add(chunks, add(0x20, mul(i, 0x20))))
            // Reads the second element (pointer) of the Content structure
            pointer := mload(add(chunk, 0x20))
            // Get size of chunk data
            size := sub(extcodesize(pointer), 1)
            // Copy chunk data from data contract
            extcodecopy(pointer, add(contents, totalSize), 1, size)
            // Update total size
            totalSize := add(totalSize, size)
        }
        // Total size is stored in the header of the return value (contents)
        mstore(contents, sub(totalSize, 0x20)) 
        // Update free memory pointer
        mstore(0x40, add(contents, and(add(totalSize, 0x1f), not(0x1f))))
    }
}

nawoo

Have you seen this code somewhere before? Actually, it is the technique used in RollerCoaster’s getLibrary function introduced in the following article.

イーサリアムnavi

Three.jsを使ったフルオンチェーンNFT「RollerCoaster」｜JavaScriptコードをPNG画像に変換しオンチェーン… 今回は、Three.jsを使ったフルオンチェーンNFTのRollerCoasterについて解説していきます。 まずは、昨年12月に投稿された、以下 xtremetom氏のツイートをご覧ください。 I …

I created NFT using EthFS.

nawoo

This time the author used EthFS to create a fully on-chain NFT.

This is a simple NFT that uploads image files to EthFS and reads and displays the images.

NFT example 1 (Uploaded on official website)

Use the image file (earth.jpg) uploaded from the official website.

This file is Base64 encoded.

The contract’s tokenURI function reads a Base64-encoded image file and sets it as the metadata image. Finally, the entire metadata is Base64-encoded and returned as dataURL.

function tokenURI(uint256 tokenId) public view virtual override returns (string memory) {
    if (!_exists(tokenId)) revert TokenDoesNotExist();
    TokenData storage token = tokenData[tokenId];
    // Create metadata
    string memory json = string.concat(
        '{"name":"',
        token.name,
        '","description":"',
        token.description,
        '","image":"data:image/jpeg;base64,',
        fileStore.getFile(token.filename).read(), // Read image files from EthFS (Base64 encoded)
        '"}'
    );
    // Base64 encoding of metadata into dataURL
    return string.concat("data:application/json;base64,", Base64.encode(bytes(json)));
}

This is all that is needed to complete the project. It was very easy to create a fully on-chain NFT.

Here is what we deployed and minted.

For the full text of the contract, please click here.

NFT example 2 (uploaded by script)

nawoo

Since Base64 encoding increases file size by 33%, it is preferable not to encode Base64 to save gas during upload.

In this case, we will try uploading an image file that is not Base64 encoded directly using the contraption.

The procedure is as follows:

• Split the file into parts of 24,575 bytes each
• Upload each part with the addContent function of the ContentStore contract to obtain the checksum
• Pass the file name, array of checksums, and metadata to the createFile function of the FileStore contract

For the actual script, please refer here.

The checksum can be obtained by using the return value of the addContent function, or by ethers.utils.keccak256() using ethers.js.

When I look at the official site, the image is not displayed, but it is uploaded without any problem.

The tokenURI function of the contract is almost the same as before, but the image file loaded from EthFS is Base64 encoded.

function tokenURI(uint256 tokenId) public view virtual override returns (string memory) {
    if (!_exists(tokenId)) revert TokenDoesNotExist();
    TokenData storage token = tokenData[tokenId];
    string memory json = string.concat(
        '{"name":"',
        token.name,
        '","description":"',
        token.description,
        '","image":"data:image/jpeg;base64,',
        Base64.encode(bytes(fileStore.getFile(token.filename).read())), // Base64 encoding of images
        '"}'
    );
    return string.concat("data:application/json;base64,", Base64.encode(bytes(json))); // Base64 encoding of metadata as well.
}

However, this contract requires that the tokenURI be Base64 encoded twice when it is created. Therefore, if the file size becomes too large, it will not be able to be processed properly due to insufficient gas required for execution.

nawoo

If the file size was about 100 KB, there was no problem.

FYI:
I would recommend Solady’s Base64.sol over OpenZeppelin’s Base64.sol as it seems to use less gas.

Here is the actual deployed and minted version.

For the full text of the contract, please click here.

Discussion and summary

Nouns DAO JAPAN aims to be the best Nouns community in the world. Click here to join the Discord.

This article explained how to use EthFS, the contents of the two contracts ContentStore and FileStore, which are the main body of EthFS, and how to actually create a fully on-chain NFT using EthFS .

We hope that this article has been helpful to those who are interested in understanding the Overview of EthFS, its usage, and the possibility of building a fully on-chain NFT.

We also encourage you to share or comment on Twitter if you find it helpful.

🆙posted an article in English about EthFS.@frolic https://t.co/xIg8egw00B

— でりおてんちょー｜derio (@yutakandori) March 18, 2023

nawoo

Having actually used EthFS this time, I found it to be quite convenient.

In terms of making Three.js and p5.js easily available to everyone, it can be used as an “on-chain version of npm” as Dom remarked, or as a place to store NFT image files instead of IPFS.

あわせて読みたい

NFTの画像データは基本Ethereumチェーン上にはありません 先日 フルオンチェーンNFT サーバー管理（ホスト）型NFT の違い、及び第3の選択肢として「IPFS」の可能性についてTwitterで言及したところ、かなり大きな反響がありまし…

However, there are a few points of concern, which are listed below.

1. Uploading from the official site results in Base64 encoding.
   • I think there are cases where Base64 encoding is not necessary depending on the application, so I would like to be able to select it.
   • According to the author, Mr. frolic, he is planning to add a checkbox for Base64 encoding in the near future.
2. Everyone is free to name the file as they wish, so the earlier the better (namespace issues).
   • If someone uploads a file named 1.jpg, others cannot name the file the same
   • Since anyone can freely upload files, there will probably be a file name batting problem in the near future.
   • This is mentioned in an issue on GitHub, so I expect that some countermeasures will be taken.
     • One idea might be to allow access by hash value like IPFS.
3. The deleteFile function exists in the FileStore contract.
   • Contract owner can delete files with the deleteFile function
     • It has the onlyOwner modifier.
   • Therefore, there is some concern about file permanence.
   • If Three.js is removed from EthFS in the future, any NFT that uses it will stop working
   • On the other hand, it may be necessary for the contract owner to remove the files, for example, when illegal files are uploaded, which is difficult to do.
   • However, deleting a file only deletes the “mapping of file name=>checksum of File structure” in the FileStore contract, but does not delete the data of the file itself.
     • File data and File structures are stored in SSTORE2 and cannot be deleted.
   • If you are concerned about your files being deleted, you could, for example, deploy a FileStore contract exclusively for yourself.

Although there are still some areas under development, we expect to see more and more projects using EthFS in the future. We may even see some unexpected use cases. I am very much looking forward to the future.

nawoo

I’m paranoid that eventually it will be standardized and we will be able to write like ethfs://~ lol

Finally, the contracts and scripts created in this article are available on GitHub here, so please refer to them.

In addition, the Earth and Moon images (CC0) were downloaded from the following site: (in Japanese only):
https://www.pexels.com/photo/earth-wallpaper-41953/
https://www.pexels.com/photo/photo-of-moon-47367/