timescaledb 0.2.1 → 0.2.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d1fef5df000abb1cc12f56397a1e18f452bf35c061260cf403809db25954451a
-  data.tar.gz: d00d7d94a33b52475a6e5415e7bab669be079f62b165abdb666764a907e34dc6
+  metadata.gz: 0367f44853d1cd4845a905e4692691baca381b739f9fb35f6a2aa471f350c946
+  data.tar.gz: b6bd9df57b80f6570f341f4842949092b367da03c5e9ba2edae6df6826288729
 SHA512:
-  metadata.gz: 1b814e74e3403c5a0e12ed8d3e115aff9a44e67cf0eff46a94b396e5511c0ce7a316808b1cc563890e27892383e1d0c3358c97abb8f9f49577bc5c8b01fc3fe6
-  data.tar.gz: ac250f31153249745e7db4ce3b7d7bbacaa25934dd7915a5a4b2aaf02595b5332344b890d1ee7a759b193f75c554bf101b7e0c492104f539edf7fa094fba133e
+  metadata.gz: 4e02cd458d020baeaa3658c20f6b502970f36772ef10f62162ad1028ad3ef7ab36943909815d3d6d04776d6cbbd8047f4705bfacbcc9315b89adaf516e54365c
+  data.tar.gz: 83f647f7814cf797155f599a3403e6641ea09edf6334f49f65279bed90c290686554b533eab852345367a0c76cf5f9a88d318d419fa46c9534aebb28a8922858

data/Fastfile

@@ -0,0 +1,17 @@
+
+# Use `fast .version_up` to rewrite the version file
+Fast.shortcut :version_up do
+  rewrite_file('(casgn nil VERSION (str _)', 'lib/timescaledb/version.rb') do |node|
+    target = node.children.last.loc.expression
+    pieces = target.source.split('.').map(&:to_i)
+    pieces.reverse.each_with_index do |fragment, i|
+      if fragment < 9
+        pieces[-(i + 1)] = fragment + 1
+        break
+      else
+        pieces[-(i + 1)] = 0
+      end
+    end
+    replace(target, "'#{pieces.join('.')}'")
+  end
+end

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    timescaledb (0.2.1)
+    timescaledb (0.2.3)
       activerecord
       activesupport
       pg (~> 1.2)
@@ -33,7 +33,7 @@ GEM
       concurrent-ruby (~> 1.0)
     method_source (1.0.0)
     minitest (5.14.4)
-    pg (1.3.1)
+    pg (1.4.4)
     pry (0.14.1)
       coderay (~> 1.1)
       method_source (~> 1.0)

data/README.md

@@ -29,7 +29,7 @@ tsdb postgres://<user>@localhost:5432/<dbname> --stats
 Or just check the stats:
 
 ```bash
-tsdb "postgres://jonatasdp@localhost:5432/timescaledb_test" --stats
+tsdb "postgres://<user>@localhost:5432/timescaledb_test" --stats
 ```
 
 These is a sample output from database example with almost no data:
@@ -53,13 +53,13 @@ as the example of database.
 
 
 ```bash
-psql postgres://jonatasdp@localhost:5432/playground -f caggs.sql
+psql postgres://<user>@localhost:5432/playground -f caggs.sql
 ```
 
 Then use `tsdb` in the command line with the same URI and `--stats`:
 
 ```bash
-tsdb postgres://jonatasdp@localhost:5432/playground --stats
+tsdb postgres://<user>@localhost:5432/playground --stats
 {:hypertables=>
   {:count=>1,
    :uncompressed=>1,
@@ -80,7 +80,7 @@ the context has a hypertable named `ticks` and a view named `ohlc_1m`.
 
 
 ```ruby
-tsdb postgres://jonatasdp@localhost:5432/playground --console
+tsdb postgres://<user>@localhost:5432/playground --console
 pry(Timescale)>
 ```
 
@@ -223,6 +223,19 @@ You can check the [all_in_one.rb](examples/all_in_one/all_in_one.rb) example tha
 6. Check chunk status
 7. Decompress a chunk
 
+### Toolkit
+
+Toolkit contains a lot of extra features to analyse data more deeply directly in
+the SQL. There are a few examples in the [examples/toolkit-demo](examples/toolkit-demo)
+folder that can let you benchmark and see the differences between implementing
+the algorithm directly in Ruby or directly in SQL using the [Timescaledb
+Toolkit](https://github.com/timescale/timescaledb-toolkit) extension.
+
+For now you can benchmark and compare:
+
+1. [volatility](examples/toolkit-demo/compare_volatility.rb) algorithm.
+2. [lttb](examples/toolkit-demo/lttb/lttb_sinatra.rb) algorithm.
+
 ### Testing
 
 If you need some inspiration for how are you going to test your hypertables,
@@ -360,7 +373,17 @@ To get compression settings for all hypertables: `Timescaledb.compression_settin
 
 ### Scopes
 
-When you enable ActsAsHypertable on your model, we include a couple default scopes. They are:
+The `acts_as_hypertable` macro can be very useful to generate some extra scopes
+for you. Example of a weather condition:
+
+```ruby
+class Condition < ActiveRecord::Base
+  acts_as_hypertable time_column: "time"
+end
+```
+
+Through the [ActsAsHypertable](./lib/timescaledb/acts_as_hypertable) on the model,
+a few scopes are created based on the `time_column` argument:
 
 | Scope name             | What they return                      |
 |------------------------|---------------------------------------|
@@ -374,6 +397,16 @@ When you enable ActsAsHypertable on your model, we include a couple default scop
 
 All time-related scopes respect your application's timezone.
 
+When you enable ActsAsTimeVector on your model, we include a couple default scopes. They are:
+
+```ruby
+class Condition < ActiveRecord::Base
+  acts_as_time_vector time_column: "time",
+    value_column: "temperature",
+    segment_by: "device_id"
+end
+```
+
 ## RSpec Hooks
 
 In case you want to use TimescaleDB on a Rails environment, you may have some
@@ -422,7 +455,7 @@ You can put some postgres URI directly as a parameter of
 `tsdb`. Here is an example from the console:
 
 ```bash
-tsdb "postgres://jonatasdp@localhost:5432/timescaledb_test"
+tsdb "postgres://<user>@localhost:5432/timescaledb_test"
 ```
 
 ## More resources
@@ -442,12 +475,11 @@ You can watch all episodes here:
 
 Here is a list of functions that would be great to have:
 
-- [ ] Dump and Restore Timescale metadata - Like db/schema.rb but for Timescale configuration.
 - [ ] Add data nodes support
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/jonatas/timescale. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/jonatas/timescale/blob/master/CODE_OF_CONDUCT.md).
+Bug reports and pull requests are welcome on GitHub at https://github.com/jonatas/timescaledb. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/jonatas/timescaledb/blob/master/CODE_OF_CONDUCT.md).
 
 ## License
 
@@ -455,4 +487,4 @@ The gem is available as open source under the terms of the [MIT License](https:/
 
 ## Code of Conduct
 
-Everyone interacting in the Timescale project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/jonatas/timescale/blob/master/CODE_OF_CONDUCT.md).
+Everyone interacting in the Timescale project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/jonatas/timescaledb/blob/master/CODE_OF_CONDUCT.md).

data/bin/console

@@ -25,4 +25,4 @@ Timescaledb::Hypertable.find_each do |hypertable|
 end
 
 require "pry"
-Pry.start(Timescale)
+Pry.start(Timescaledb)

data/bin/tsdb

@@ -9,7 +9,7 @@ Timescaledb::Hypertable.find_each do |hypertable|
   class_name = hypertable.hypertable_name.singularize.camelize
   model = Class.new(ActiveRecord::Base) do
     self.table_name = hypertable.hypertable_name
-    acts_as_hypertable
+    acts_as_hypertable time_column: hypertable.main_dimension.column_name
   end
   Timescaledb.const_set(class_name, model)
 end
@@ -44,5 +44,5 @@ if ARGV.index("--stats")
 end
 
 if ARGV.index("--console")
-  Pry.start(Timescale)
+  Pry.start(Timescaledb)
 end

data/docs/command_line.md

@@ -0,0 +1,178 @@
+# Command line application
+
+When you install the gem locally, a new command line application named `tsdb` will be available on your command line.
+
+## The `tsdb` CLI
+
+It accepts a Postgresql URI and some extra flags that can help you to get more info from your TimescaleDB server:
+
+```bash
+tsdb <uri> --stats
+```
+
+Where the `<uri>` is replaced with params from your connection like:
+
+```bash
+tsdb postgres://<user>@localhost:5432/<dbname> --stats
+```
+
+Or merely check the stats:
+
+```bash
+tsdb "postgres://<user>@localhost:5432/timescaledb_test" --stats
+```
+
+Here is a sample output from a database example with almost no data:
+
+```ruby
+{:hypertables=>
+  {:count=>3,
+   :uncompressed=>2,
+   :chunks=>{:total=>1, :compressed=>0, :uncompressed=>1},
+   :size=>{:befoe_compressing=>"80 KB", :after_compressing=>"0 Bytes"}},
+ :continuous_aggregates=>{:count=>1},
+ :jobs_stats=>[{:success=>nil, :runs=>nil, :failures=>nil}]}
+```
+
+To start a interactive ruby/[pry](https://github.com/pry/pry) console use `--console`:
+The console will dynamically create models for all hypertables that it finds
+in the database.
+
+Let's consider the [caggs.sql](https://gist.github.com/jonatas/95573ad8744994094ec9f284150004f9#file-caggs-sql) as the example of a database.
+
+
+```bash
+psql postgres://<user>@localhost:5432/playground -f caggs.sql
+```
+
+Then use `tsdb` in the command line with the same URI and `--stats`:
+
+```ruby
+tsdb postgres://<user>@localhost:5432/playground --stats
+{:hypertables=>
+  {:count=>1,
+   :uncompressed=>1,
+   :approximate_row_count=>{"ticks"=>352},
+   :chunks=>{:total=>1, :compressed=>0, :uncompressed=>1},
+   :size=>{:uncompressed=>"88 KB", :compressed=>"0 Bytes"}},
+ :continuous_aggregates=>{:total=>1},
+ :jobs_stats=>[{:success=>nil, :runs=>nil, :failures=>nil}]}
+```
+
+To have some interactive playground with the actual database using ruby, just
+try the same command before changing from `--stats` to `--console`:
+
+### tsdb --console
+
+We are using the same database from the previous example for this context which contains a hypertable named `ticks` and a view called `ohlc_1m`.
+
+
+```ruby
+tsdb postgres://<user>@localhost:5432/playground --console
+pry(Timescale)>
+```
+
+The `tsdb` CLI will automatically create ActiveRecord models for hypertables and the continuous aggregates views.
+
+```ruby
+Tick
+=> Timescaledb::Tick(time: datetime, symbol: string, price: decimal, volume: integer)
+```
+
+Note that it's only created for this session and will never cache in the
+library or any other place.
+
+In this case, the `Tick` model comes from the `ticks` hypertable found in the database.
+It contains several methods inherited from the `acts_as_hypertable` macro.
+
+Let's start with the `.hypertable` method.
+
+```ruby
+Tick.hypertable
+=> #<Timescaledb::Hypertable:0x00007fe99c258900
+ hypertable_schema: "public",
+ hypertable_name: "ticks",
+ owner: "jonatasdp",
+ num_dimensions: 1,
+ num_chunks: 1,
+ compression_enabled: false,
+ is_distributed: false,
+ replication_factor: nil,
+ data_nodes: nil,
+ tablespaces: nil>
+```
+
+The core of the hypertables is the fragmentation of the data into chunks, the child tables that distribute the data. You can check all chunks directly from the hypertable relation.
+
+```ruby
+Tick.hypertable.chunks
+unknown OID 2206: failed to recognize type of 'primary_dimension_type'. It will cast as a String.
+=> [#<Timescaledb::Chunk:0x00007fe99c31b068
+  hypertable_schema: "public",
+  hypertable_name: "ticks",
+  chunk_schema: "_timescaledb_internal",
+  chunk_name: "_hyper_33_17_chunk",
+  primary_dimension: "time",
+  primary_dimension_type: "timestamp without time zone",
+  range_start: 1999-12-30 00:00:00 +0000,
+  range_end: 2000-01-06 00:00:00 +0000,
+  range_start_integer: nil,
+  range_end_integer: nil,
+  is_compressed: false,
+  chunk_tablespace: nil,
+  data_nodes: nil>]
+```
+
+> Chunks are created by partitioning the hypertable data into one
+> (or potentially multiple) dimensions. All hypertables are partitions by the
+> values belonging to a time column, which may be in timestamp, date, or
+> various integer forms. If the time partitioning interval is one day,
+> for example, then rows with timestamps that belong to the same day are co-located
+> within the same chunk, while rows belonging to different days belong to different chunks.
+> Learn more [here](https://docs.timescale.com/timescaledb/latest/overview/core-concepts/hypertables-and-chunks/).
+
+Another core concept of TimescaleDB is compression. With data partitioned, it
+becomes very convenient to compress and decompress chunks independently.
+
+```ruby
+Tick.hypertable.chunks.first.compress!
+ActiveRecord::StatementInvalid: PG::FeatureNotSupported: ERROR:  compression not enabled on "ticks"
+DETAIL:  It is not possible to compress chunks on a hypertable that does not have compression enabled.
+HINT:  Enable compression using ALTER TABLE with the timescaledb.compress option.
+```
+
+As compression is not enabled, let's do it by executing plain SQL directly from the actual context. To borrow a connection, let's use the Tick object.
+
+```ruby
+Tick.connection.execute("ALTER TABLE ticks SET (timescaledb.compress)") # => PG_OK
+```
+
+And now, it's possible to compress and decompress:
+
+```ruby
+Tick.hypertable.chunks.first.compress!
+Tick.hypertable.chunks.first.decompress!
+```
+Learn more about TimescaleDB compression [here](https://docs.timescale.com/timescaledb/latest/overview/core-concepts/compression/).
+
+The `ohlc_1m` view is also available as an ActiveRecord:
+
+```ruby
+Ohlc1m
+=> Timescaledb::Ohlc1m(bucket: datetime, symbol: string, open: decimal, high: decimal, low: decimal, close: decimal, volume: integer)
+```
+
+And you can run any query as you do with regular active record queries.
+
+```ruby
+Ohlc1m.order(bucket: :desc).last
+=> #<Timescaledb::Ohlc1m:0x00007fe99c2c38e0
+ bucket: 2000-01-01 00:00:00 UTC,
+ symbol: "SYMBOL",
+ open: 0.13e2,
+ high: 0.3e2,
+ low: 0.1e1,
+ close: 0.1e2,
+ volume: 27600>
+```
+

data/docs/index.md

@@ -0,0 +1,61 @@
+# The TimescaleDB Ruby Gem
+
+Welcome to the TimescaleDB gem! To experiment with the code, start installing the
+gem:
+
+## Installing
+
+You can install the gem locally:
+
+```bash
+gem install timescaledb
+```
+
+Or require it directly in the Gemfile of your project:
+
+```ruby
+gem "timescaledb"
+```
+
+## Features
+
+* The model can use the [acts_as_hypertable](https://github.com/jonatas/timescaledb/tree/master/lib/timescaledb/acts_as_hypertable.rb) macro. Check more on [models](models) documentation.
+* The ActiveRecord [migrations](migrations) can use the [create_table](https://github.com/jonatas/timescaledb/tree/master/lib/timescaledb/migration_helpers.rb) supporting the `hypertable` keyword. It's also enabling you to add retention and continuous aggregates policies
+* A standalone `create_hypertable` macro is also allowed in the migrations.
+* Testing also becomes easier as the [schema dumper](https://github.com/jonatas/timescaledb/tree/master/lib/timescaledb/schema_dumper.rb) will automatically introduce the hypertables to all environments.
+* It also contains a [scenic extension](https://github.com/jonatas/timescaledb/tree/master/lib/timescaledb/scenic/extension.rb) to work with [scenic views](https://github.com/scenic-views/scenic) as it's a wide adoption in the community.
+* The gem is also packed with a [command line utility](command_line) that makes it easier to navigate in your database with Pry and all your hypertables available in a Ruby style.
+
+## Examples
+
+The [all_in_one](https://github.com/jonatas/timescaledb/tree/master/examples/all_in_one/all_in_one.rb) example shows:
+
+1. Create a hypertable with compression settings
+2. Insert data
+3. Run some queries
+4. Check chunk size per model
+5. Compress a chunk
+6. Check chunk status
+7. Decompress a chunk
+
+The [ranking](https://github.com/jonatas/timescaledb/tree/master/examples/ranking) example shows how to configure a Rails app and navigate all the features available.
+
+## Extra resources
+
+If you need extra help, please join the fantastic [timescale community](https://www.timescale.com/community)
+or ask your question on [StackOverflow](https://stackoverflow.com/questions/tagged/timescaledb) using the `#timescaledb` tag.
+
+If you want to go deeper in the library, the [videos](videos) links to all
+live-coding sessions showed how [@jonatasdp](https://twitter.com/jonatasdp) built the gem.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/jonatas/timescaledb. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/jonatas/timescaledb/blob/master/CODE_OF_CONDUCT.md).
+
+## License
+
+The gem is available as open source under the [MIT License](https://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in the Timescale project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the [code of conduct](https://github.com/jonatas/timescaledb/blob/master/CODE_OF_CONDUCT.md).

data/docs/migrations.md

@@ -0,0 +1,69 @@
+# ActiveRecord migrations helpers for Timescale
+
+Create table is now with the `hypertable` keyword allowing to pass a few options
+to the function call while also using the `create_table` method:
+
+## create_table with the `:hypertable` option
+
+```ruby
+hypertable_options = {
+  time_column: 'created_at',
+  chunk_time_interval: '1 min',
+  compress_segmentby: 'identifier',
+  compression_interval: '7 days'
+}
+
+create_table(:events, id: false, hypertable: hypertable_options) do |t|
+  t.string :identifier, null: false
+  t.jsonb :payload
+  t.timestamps
+end
+```
+
+## The `create_continuous_aggregate` helper
+
+This example shows a ticks table grouping ticks as OHLCV histograms for every
+minute.
+
+```ruby
+hypertable_options = {
+  time_column: 'created_at',
+  chunk_time_interval: '1 min',
+  compress_segmentby: 'symbol',
+  compress_orderby: 'created_at',
+  compression_interval: '7 days'
+}
+create_table :ticks, hypertable: hypertable_options, id: false do |t|
+  t.string :symbol
+  t.decimal :price
+  t.integer :volume
+  t.timestamps
+end
+Tick = Class.new(ActiveRecord::Base) do
+  self.table_name = 'ticks'
+  self.primary_key = 'symbol'
+  acts_as_hypertable
+end
+
+query = Tick.select(<<~QUERY)
+  time_bucket('1m', created_at) as time,
+  symbol,
+  FIRST(price, created_at) as open,
+  MAX(price) as high,
+  MIN(price) as low,
+  LAST(price, created_at) as close,
+  SUM(volume) as volume").group("1,2")
+QUERY
+
+options = {
+  with_data: false,
+  refresh_policies: {
+    start_offset: "INTERVAL '1 month'",
+    end_offset: "INTERVAL '1 minute'",
+    schedule_interval: "INTERVAL '1 minute'"
+  }
+}
+
+create_continuous_aggregate('ohlc_1m', query, **options)
+```
+

data/docs/models.md

@@ -0,0 +1,78 @@
+# Models
+
+The ActiveRecord is the default ORM in the Ruby community. We have introduced a macro that helps you to inject the behavior as other libraries do in the Rails ecosystem.
+
+## The `acts_as_hypertable` macro
+
+You can declare a Rails model as a Hypertable by invoking the `acts_as_hypertable` macro. This macro extends your existing model with timescaledb-related functionality.
+model:
+
+```ruby
+class Event < ActiveRecord::Base
+  acts_as_hypertable
+end
+```
+
+By default, ActsAsHypertable assumes a record's _time_column_ is called `created_at`.
+
+### Options
+
+If you are using a different time_column name, you can specify it as follows when invoking the `acts_as_hypertable` macro:
+
+```ruby
+class Event < ActiveRecord::Base
+  acts_as_hypertable time_column :timestamp
+end
+```
+
+### Chunks
+
+To get all the chunks from a model's hypertable, you can use `.chunks`.
+
+```ruby
+Event.chunks # => [#<Timescaledb::Chunk>, ...]
+```
+
+### Hypertable metadata
+
+To get the models' hypertable metadata, you can use `.hypertable`.
+
+```ruby
+Event.hypertable # => #<Timescaledb::Hypertable>
+```
+
+To get hypertable metadata for all hypertables: `Timescaledb.hypertables`.
+
+### Compression Settings
+
+Compression settings are accessible through the hypertable.
+
+```ruby
+Event.hypertable.compression_settings # => [#<Timescaledb::CompressionSettings>, ...]
+```
+
+To get compression settings for all hypertables: `Timescaledb.compression_settings`.
+
+### Scopes
+
+When you enable ActsAsHypertable on your model, we include a few default scopes. They are:
+
+| Scope name             | What they return                      |
+|------------------------|---------------------------------------|
+| `Model.previous_month` | Records created in the previous month |
+| `Model.previous_week`  | Records created in the previous week  |
+| `Model.this_month`     | Records created this month            |
+| `Model.this_week`      | Records created this week             |
+| `Model.yesterday`      | Records created yesterday             |
+| `Model.today`          | Records created today                 |
+| `Model.last_hour`      | Records created in the last hour      |
+
+All time-related scopes respect your application's timezone.
+
+
+## Scenic integration
+
+The [Scenic](https://github.com/scenic-views/scenic) gem is easy to
+manage database view definitions for a Rails application. Unfortunately, TimescaleDB's continuous aggregates are more complex than regular PostgreSQL views, and the schema dumper included with Scenic can't dump a complete definition.
+
+This gem automatically configures Scenic to use a `Timescaledb::Scenic::Adapter.` which will correctly handle schema dumping.