rocksdb-native 0.0.0 → 0.0.1

package/vendor/rocksdb/docs/_posts/2015-07-15-rocksdb-2015-h2-roadmap.markdown

@@ -1,92 +0,0 @@
----
-title: RocksDB 2015 H2 roadmap
-layout: post
-author: icanadi
-category: blog
-redirect_from:
-  - /blog/2015/rocksdb-2015-h2-roadmap/
----
-
-Every 6 months, RocksDB team gets together to prioritize the work ahead of us. We just went through this exercise and we wanted to share the results with the community. Here's what RocksDB team will be focusing on for the next 6 months:
-
-<!--truncate-->
-
-**MyRocks**
-
-As you might know, we're working hard to integrate RocksDB as a storage engine for MySQL. This project is pretty important for us because we're heavy users of MySQL. We're already getting pretty good performance results, but there is more work to be done. We need to focus on both performance and stability. The most high priority items on are list are:
-
-
-
-
-  1. Reduce CPU costs of RocksDB as a MySQL storage engine
-
-
-  2. Implement pessimistic concurrency control to support repeatable read isolation level in MyRocks
-
-
-  3. Reduce P99 read latency, which is high mostly because of lingering tombstones
-
-
-  4. Port ZSTD compression
-
-
-**MongoRocks**
-
-Another database that we're working on is MongoDB. The project of integrating MongoDB with RocksDB storage engine is called MongoRocks. It's already running in production at Parse [1] and we're seeing surprisingly few issues. Our plans for the next half:
-
-
-
-
-  1. Keep improving performance and stability, possibly reuse work done on MyRocks (workloads are pretty similar).
-
-
-  2. Increase internal and external adoption.
-
-
-  3. Support new MongoDB 3.2.
-
-
-**RocksDB on cheaper storage media**
-
-Up to now, our mission was to build the best key-value store “for fast storage” (flash and in-memory). However, there are some use-cases at Facebook that don't need expensive high-end storage. In the next six months, we plan to deploy RocksDB on cheaper storage media. We will optimize performance to RocksDB on either or both:
-
-
-
-
-  1. Hard drive storage array.
-
-
-  2. Tiered Storage.
-
-
-**Quality of Service**
-
-When talking to our customers, there are couple of issues that keep reoccurring. We need to fix them to make our customers happy. We will improve RocksDB to provide better assurance of performance and resource usage. Non-exhaustive list includes:
-
-
-
-
-  1. Iterate P99 can be high due to the presence of tombstones.
-
-
-  2. Write stalls can happen during high write loads.
-
-
-  3. Better control of memory and disk usage.
-
-
-  4. Service quality and performance of backup engine.
-
-
-**Operation's user experience**
-
-As we increase deployment of RocksDB, engineers are spending more time on debugging RocksDB issues. We plan to improve user experience when running RocksDB. The goal is to reduce TTD (time-to-debug). The work includes monitoring, visualizations and documentations.
-
-[1]( http://blog.parse.com/announcements/mongodb-rocksdb-parse/](http://blog.parse.com/announcements/mongodb-rocksdb-parse/)
-
-
-### Comments
-
-**[Mike](allspace2012@outlook.com)**
-
-What’s the status of this roadmap? “RocksDB on cheaper storage media”, has this been implemented?

package/vendor/rocksdb/docs/_posts/2015-07-17-spatial-indexing-in-rocksdb.markdown

@@ -1,78 +0,0 @@
----
-title: Spatial indexing in RocksDB
-layout: post
-author: icanadi
-category: blog
-redirect_from:
-  - /blog/2039/spatial-indexing-in-rocksdb/
----
-
-About a year ago, there was a need to develop a spatial database at Facebook. We needed to store and index Earth's map data. Before building our own, we looked at the existing spatial databases. They were all very good technology, but also general purpose. We could sacrifice a general-purpose API, so we thought we could build a more performant database, since it would be specifically designed for our use-case. Furthermore, we decided to build the spatial database on top of RocksDB, because we have a lot of operational experience with running and tuning RocksDB at a large scale.
-
-<!--truncate-->
-
-When we started looking at this project, the first thing that surprised us was that our planet is not that big. Earth's entire map data can fit in memory on a reasonably high-end machine. Thus, we also decided to build a spatial database optimized for memory-resident dataset.
-
-The first use-case of our spatial database was an experimental map renderer. As part of our project, we successfully loaded [Open Street Maps](https://www.openstreetmap.org/) dataset and hooked it up with [Mapnik](http://mapnik.org/), a map rendering engine.
-
-The usual Mapnik workflow is to load the map data into a SQL-based database and then define map layers with SQL statements. To render a tile, Mapnik needs to execute a couple of SQL queries. The benefit of this approach is that you don't need to reload your database when you change your map style. You can just change your SQL query and Mapnik picks it up. In our model, we decided to precompute the features we need for each tile. We need to know the map style before we create the database. However, when rendering the map tile, we only fetch the features that we need to render.
-
-We haven't open sourced the RocksDB Mapnik plugin or the database loading pipeline. However, the spatial indexing is available in RocksDB under a name [SpatialDB](https://github.com/facebook/rocksdb/blob/main/include/rocksdb/utilities/spatial_db.h). The API is focused on map rendering use-case, but we hope that it can also be used for other spatial-based applications.
-
-Let's take a tour of the API. When you create a spatial database, you specify the spatial indexes that need to be built. Each spatial index is defined by a bounding box and granularity. For map rendering, we create a spatial index for each zoom levels. Higher zoom levels have more granularity.
-
-
-
-    SpatialDB::Create(
-      SpatialDBOptions(),
-      "/data/map", {
-        SpatialIndexOptions("zoom10", BoundingBox(0, 0, 100, 100), 10),
-        SpatialIndexOptions("zoom16", BoundingBox(0, 0, 100, 100), 16)
-      }
-    );
-
-
-
-
-When you insert a feature (building, street, country border) into SpatialDB, you need to specify the list of spatial indexes that will index the feature. In the loading phase we process the map style to determine the list of zoom levels on which we'll render the feature. For example, we will not render the building on zoom level that shows an entire country. Building will only be indexed on higher zoom level's index. Country borders will be indexes on all zoom levels.
-
-
-
-    FeatureSet feature;
-    feature.Set("type", "building");
-    feature.Set("height", 6);
-    db->Insert(WriteOptions(), BoundingBox<double>(5, 5, 10, 10),
-               well_known_binary_blob, feature, {"zoom16"});
-
-
-
-
-The indexing part is pretty simple. For each feature, we first find a list of index tiles that it intersects. Then, we add a link from the tile's [quad key](https://msdn.microsoft.com/en-us/library/bb259689.aspx) to the feature's primary key. Using quad keys improves data locality, i.e. features closer together geographically will have similar quad keys. Even though we're optimizing for a memory-resident dataset, data locality is still very important due to different caching effects.
-
-After you're done inserting all the features, you can call an API Compact() that will compact the dataset and speed up read queries.
-
-
-
-    db->Compact();
-
-
-
-
-SpatialDB's query specifies: 1) bounding box we're interested in, and 2) a zoom level. We find all tiles that intersect with the query's bounding box and return all features in those tiles.
-
-
-
-
-    Cursor* c = db_->Query(ReadOptions(), BoundingBox<double>(1, 1, 7, 7), "zoom16");
-    for (c->Valid(); c->Next()) {
-        Render(c->blob(), c->feature_set());
-    }
-
-
-
-
-Note: `Render()` function is not part of RocksDB. You will need to use one of many open source map renderers, for example check out [Mapnik](http://mapnik.org/).
-
-TL;DR If you need an embedded spatial database, check out RocksDB's SpatialDB. [Let us know](https://www.facebook.com/groups/rocksdb.dev/) how we can make it better.
-
-If you're interested in learning more, check out this [talk](https://www.youtube.com/watch?v=T1jWsDMONM8).

package/vendor/rocksdb/docs/_posts/2015-07-22-rocksdb-is-now-available-in-windows-platform.markdown

@@ -1,30 +0,0 @@
----
-title: RocksDB is now available in Windows Platform
-layout: post
-author: dmitrism
-category: blog
-redirect_from:
-  - /blog/2033/rocksdb-is-now-available-in-windows-platform/
----
-
-Over the past 6 months we have seen a number of use cases where RocksDB is successfully used by the community and various companies to achieve high throughput and volume in a modern server environment.
-
-We at Microsoft Bing could not be left behind. As a result we are happy to [announce](http://bit.ly/1OmWBT9) the availability of the Windows Port created here at Microsoft which we intend to use as a storage option for one of our key/value data stores.
-
-<!--truncate-->
-
-We are happy to make this available for the community. Keep tuned for more announcements to come.
-
-### Comments
-
-**[Siying Dong](siying.d@fb.com)**
-
-Appreciate your contributions to RocksDB project! I believe it will benefits many users!
-
-**[empresas sevilla](oxofkx@gmail.com)**
-
-Magnifico artículo|, un placer leer el blog
-
-**[jak usunac](tomogedac@o2.pl)**
-
-I believe it will benefits too

package/vendor/rocksdb/docs/_posts/2015-07-23-dynamic-level.markdown

@@ -1,29 +0,0 @@
----
-title: Dynamic Level Size for Level-Based Compaction
-layout: post
-author: sdong
-category: blog
-redirect_from:
-  - /blog/2207/dynamic-level/
----
-
-In this article, we follow up on the first part of an answer to one of the questions in our [AMA](https://www.reddit.com/r/IAmA/comments/3de3cv/we_are_rocksdb_engineering_team_ask_us_anything/ct4a8tb), the dynamic level size in level-based compaction.
-
-<!--truncate-->
-
-Level-based compaction is the original LevelDB compaction style and one of the two major compaction styles in RocksDB (See [our wiki](https://github.com/facebook/rocksdb/wiki/RocksDB-Basics#multi-threaded-compactions)). In RocksDB we introduced parallelism and more configurable options to it but the main algorithm stayed the same, until we recently introduced the dynamic level size mode.
-
-
-In level-based compaction, we organize data to different sorted runs, called levels. Each level has a target size.  Usually target size of levels increases by the same size multiplier. For example, you can set target size of level 1 to be 1GB, and size multiplier to be 10, and the target size of level 1, 2, 3, 4 will be 1GB, 10GB, 100GB and 1000GB. Before level 1, there will be some staging file flushed from mem tables, called Level 0 files, which will later be merged to level 1. Compactions will be triggered as soon as actual size of a level exceeds its target size. We will merge a subset of data of that level to next level, to reduce size of the level. More compactions will be triggered until sizes of all the levels are lower than their target sizes. In a steady state, the size of each level will be around the same size of the size of level targets.
-
-
-Level-based compaction’s advantage is its good space efficiency. We usually use the metric space amplification to measure the space efficiency. In this article ignore the effects of data compression so space amplification= size_on_file_system / size_of_user_data.
-
-
-How do we estimate space amplification of level-based compaction? We focus specifically on the databases in steady state, which means database size is stable or grows slowly over time. This means updates will add roughly the same or little more data than what is removed by deletes. Given that, if we compact all the data all to the last level, the size of level will be equal as the size of last level before the compaction. On the other hand, the size of user data will be approximately the size of DB if we compact all the levels down to the last level. So the size of the last level will be a good estimation of user data size. So total size of the DB divided by the size of the last level will be a good estimation of space amplification.
-
-
-Applying the equation, if we have four non-zero levels, their sizes are 1GB, 10GB, 100GB, 1000GB, the size amplification will be approximately (1000GB + 100GB + 10GB + 1GB) / 1000GB = 1.111, which is a very good number. However, there is a catch here: how to make sure the last level’s size is 1000GB, the same as the level’s size target? A user has to fine tune level sizes to achieve this number and will need to re-tune if DB size changes. The theoretic number 1.11 is hard to achieve in practice. In a worse case, if you have the target size of last level to be 1000GB but the user data is only 200GB, then the actual space amplification will be (200GB + 100GB + 10GB + 1GB) / 200GB = 1.555, a much worse number.
-
-
-To solve this problem, my colleague Igor Kabiljo came up with a solution of dynamic level size target mode. You can enable it by setting options.level_compaction_dynamic_level_bytes=true. In this mode, size target of levels are changed dynamically based on size of the last level. Suppose the level size multiplier to be 10, and the DB size is 200GB. The target size of the last level is automatically set to be the actual size of the level, which is 200GB, the second to last level’s size target will be automatically set to be size_last_level / 10 = 20GB, the third last level’s will be size_last_level/100 = 2GB, and next level to be size_last_level/1000 = 200MB. We stop here because 200MB is within the range of the first level. In this way, we can achieve the 1.111 space amplification, without fine tuning of the level size targets. More details can be found in [code comments of the option](https://github.com/facebook/rocksdb/blob/v3.11/include/rocksdb/options.h#L366-L423) in the header file.

package/vendor/rocksdb/docs/_posts/2015-10-27-getthreadlist.markdown

@@ -1,193 +0,0 @@
----
-title: GetThreadList
-layout: post
-author: yhciang
-category: blog
-redirect_from:
-  - /blog/2261/getthreadlist/
----
-
-We recently added a new API, called `GetThreadList()`, that exposes the RocksDB background thread activity. With this feature, developers will be able to obtain the real-time information about the currently running compactions and flushes such as the input / output size, elapsed time, the number of bytes it has written. Below is an example output of `GetThreadList`.  To better illustrate the example, we have put a sample output of `GetThreadList` into a table where each column represents a thread status:
-
-<!--truncate-->
-
-<table width="637" >
-<tbody >
-<tr style="border:2px solid #000000" >
-
-<td style="padding:3px" >ThreadID
-</td>
-
-<td >140716395198208
-</td>
-
-<td >140716416169728
-</td>
-</tr>
-<tr >
-
-<td style="padding:3px" >DB
-</td>
-
-<td >db1
-</td>
-
-<td >db2
-</td>
-</tr>
-<tr >
-
-<td style="padding:3px" >CF
-</td>
-
-<td >default
-</td>
-
-<td >picachu
-</td>
-</tr>
-<tr >
-
-<td style="padding:3px" >ThreadType
-</td>
-
-<td >High Pri
-</td>
-
-<td >Low Pri
-</td>
-</tr>
-<tr >
-
-<td style="padding:3px" >Operation
-</td>
-
-<td >Flush
-</td>
-
-<td >Compaction
-</td>
-</tr>
-<tr >
-
-<td style="padding:3px" >ElapsedTime
-</td>
-
-<td >143.459 ms
-</td>
-
-<td >607.538 ms
-</td>
-</tr>
-<tr >
-
-<td style="padding:3px" >Stage
-</td>
-
-<td >FlushJob::WriteLevel0Table
-</td>
-
-<td >CompactionJob::Install
-</td>
-</tr>
-<tr >
-
-<td style="vertical-align:top;padding:3px" >OperationProperties
-</td>
-
-<td style="vertical-align:top;padding:3px" >
-BytesMemtables 4092938
-BytesWritten 1050701
-</td>
-
-<td style="vertical-align:top" >
-BaseInputLevel 1
-BytesRead 4876417
-BytesWritten 4140109
-IsDeletion 0
-IsManual 0
-IsTrivialMove 0
-JobID 146
-OutputLevel 2
-TotalInputBytes 4883044
-</td>
-</tr>
-</tbody>
-</table>
-
-In the above output, we can see `GetThreadList()` reports the activity of two threads: one thread running flush job (middle column) and the other thread running a compaction job (right-most column).  In each thread status, it shows basic information about the thread such as thread id, it's target db / column family, and the job it is currently doing and the current status of the job.  For instance, we can see thread 140716416169728 is doing compaction on the `picachu` column family in database `db2`.  In addition, we can see the compaction has been running for 600 ms, and it has read 4876417 bytes out of 4883044 bytes. This indicates the compaction is about to complete.  The stage property indicates which code block the thread is currently executing.  For instance, thread 140716416169728 is currently running `CompactionJob::Install`, which further indicates the compaction job is almost done.
-
-Below we briefly describe its API.
-
-
-## How to Enable it?
-
-
-To enable thread-tracking of a rocksdb instance, simply set `enable_thread_tracking` to true in its DBOptions:
-
-```c++
-// If true, then the status of the threads involved in this DB will
-// be tracked and available via GetThreadList() API.
-//
-// Default: false
-bool enable_thread_tracking;
-```
-
-
-
-## The API
-
-
-The GetThreadList API is defined in [include/rocksdb/env.h](https://github.com/facebook/rocksdb/blob/main/include/rocksdb/env.h#L317-L318), which is an Env
-function:
-
-```c++
-virtual Status GetThreadList(std::vector* thread_list)
-```
-
-Since an Env can be shared across multiple rocksdb instances, the output of
-`GetThreadList()` include the background activity of all the rocksdb instances
-that using the same Env.
-
-The `GetThreadList()` API simply returns a vector of `ThreadStatus`, each describes
-the current status of a thread. The `ThreadStatus` structure, defined in
-[include/rocksdb/thread_status.h](https://github.com/facebook/rocksdb/blob/main/include/rocksdb/thread_status.h), contains the following information:
-
-```c++
-// An unique ID for the thread.
-const uint64_t thread_id;
-
-// The type of the thread, it could be HIGH_PRIORITY,
-// LOW_PRIORITY, and USER
-const ThreadType thread_type;
-
-// The name of the DB instance where the thread is currently
-// involved with. It would be set to empty string if the thread
-// does not involve in any DB operation.
-const std::string db_name;
-
-// The name of the column family where the thread is currently
-// It would be set to empty string if the thread does not involve
-// in any column family.
-const std::string cf_name;
-
-// The operation (high-level action) that the current thread is involved.
-const OperationType operation_type;
-
-// The elapsed time in micros of the current thread operation.
-const uint64_t op_elapsed_micros;
-
-// An integer showing the current stage where the thread is involved
-// in the current operation.
-const OperationStage operation_stage;
-
-// A list of properties that describe some details about the current
-// operation. Same field in op_properties[] might have different
-// meanings for different operations.
-uint64_t op_properties[kNumOperationProperties];
-
-// The state (lower-level action) that the current thread is involved.
-const StateType state_type;
-```
-
-If you are interested in the background thread activity of your RocksDB application, please feel free to give `GetThreadList()` a try :)

package/vendor/rocksdb/docs/_posts/2015-11-10-use-checkpoints-for-efficient-snapshots.markdown

@@ -1,45 +0,0 @@
----
-title: Use Checkpoints for Efficient Snapshots
-layout: post
-author: rven2
-category: blog
-redirect_from:
-  - /blog/2609/use-checkpoints-for-efficient-snapshots/
----
-
-**Checkpoint** is a feature in RocksDB which provides the ability to take a snapshot of a running RocksDB database in a separate directory. Checkpoints can be used as a point in time snapshot, which can be opened Read-only to query rows as of the point in time or as a Writeable snapshot by opening it Read-Write. Checkpoints can be used for both full and incremental backups.
-
-<!--truncate-->
-
-
-The Checkpoint feature enables RocksDB to create a consistent snapshot of a given RocksDB database in the specified directory. If the snapshot is on the same filesystem as the original database, the SST files will be hard-linked, otherwise SST files will be copied. The manifest and CURRENT files will be copied. In addition, if there are multiple column families, log files will be copied for the period covering the start and end of the checkpoint, in order to provide a consistent snapshot across column families.
-
-
-
-
-A Checkpoint object needs to be created for a database before checkpoints are created. The API is as follows:
-
-
-
-
-`Status Create(DB* db, Checkpoint** checkpoint_ptr);`
-
-
-
-
-Given a checkpoint object and a directory, the CreateCheckpoint function creates a consistent snapshot of the database in the given directory.
-
-
-
-
-`Status CreateCheckpoint(const std::string& checkpoint_dir);`
-
-
-
-
-The directory should not already exist and will be created by this API. The directory will be an absolute path. The checkpoint can be used as a ​read-only copy of the DB or can be opened as a standalone DB. When opened read/write, the SST files continue to be hard links and these links are removed when the files are obsoleted. When the user is done with the snapshot, the user can delete the directory to remove the snapshot.
-
-
-
-
-Checkpoints are used for online backup in ​MyRocks. which is MySQL using RocksDB as the storage engine . ([MySQL on RocksDB](https://github.com/facebook/mysql-5.6)) ​