observability 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9e8e5db4293106c0b48ed3ea428c4a786eec948d11f19e8a78950ee49383655b
+  data.tar.gz: a9195f349e14fbc0721c22af67f8cb115bb15264a68dbf12ffd4243418f62167
+SHA512:
+  metadata.gz: a851f201c9ff6757fb53bd14455b062fec2e4923d1386ee0192bc7f3033b3bb74ce48549b2b21ce4888ed674cb6f48050c89039b9d678d32baf85721fde965db
+  data.tar.gz: b7eee539d2e8a7ea9ae40262071b21f986945786f2409e3721342e182e5acb6bee27c25a2dc35ae5db1952aa434c32faec23203946b345cdfb88201fc7f4b4f9

data/.document

@@ -0,0 +1,5 @@
+lib/**/*.rb
+README.md
+ChangeLog.md
+History.md
+LICENSE.txt

data/.rdoc_options

@@ -0,0 +1,16 @@
+--- !ruby/object:RDoc::Options
+encoding: UTF-8
+static_path: []
+rdoc_include:
+  - .
+charset: UTF-8
+exclude: 
+hyperlink_all: false
+line_numbers: false
+main_page: README.md
+markup: markdown
+show_hash: false
+tab_width: 8
+title: Observability Documentation
+visibility: :protected
+webcvs:

data/.simplecov

@@ -0,0 +1,9 @@
+# Simplecov config
+
+SimpleCov.start do
+	add_filter 'spec'
+	add_filter 'integration'
+	add_group "Needing tests" do |file|
+		file.covered_percent < 90
+	end
+end

data/ChangeLog

@@ -0,0 +1,139 @@
+2019-07-22  Michael Granger  <ged@FaerieMUD.org>
+
+@  	* .hgignore, DevNotes.md, Manifest.txt, lib/observability.rb,
+|  	lib/observability/event.rb, lib/observability/observer.rb,
+|  	lib/observability/observer_hooks.rb, lib/observability/sender.rb,
+|  	lib/observability/sender/logger.rb,
+|  	lib/observability/sender/testing.rb,
+|  	lib/observability/sender/udp.rb, spec/observability/event_spec.rb,
+|  	spec/observability/observer_hooks_spec.rb,
+|  	spec/observability/observer_spec.rb,
+|  	spec/observability/sender/udp_spec.rb,
+|  	spec/observability/sender_spec.rb, spec/observability_spec.rb:
+|  	Add a bunch more work:
+|
+|  	- Add the beginnings of context
+|  	- Refactor the sender to be more easily subclassed
+|  	- Add a UDP sender
+|  	- Refactor finishing around a block up to the Observer
+|  	- Rename `observer` hook to `observability` to avoid colliding with
+|  	observer-pattern systems.
+|  	- Add pending specs for intended features
+|  	- Removed unused `observed_system` DSL method
+|  	- Concurrent logs to Observability logger
+|  	[c206c306394e] [tip]
+|
+2019-07-19  Michael Granger  <ged@FaerieMUD.org>
+
+o  	* bin/observability-collector, experiments/postgres-events.sql,
+|  	lib/observability.rb, lib/observability/collector.rb,
+|  	lib/observability/collector/timescale.rb,
+|  	lib/observability/event.rb, lib/observability/sender/logger.rb,
+|  	spec/observability/sender/logger_spec.rb,
+|  	spec/observability_spec.rb:
+|  	More work on the collector spike, start adding callbacks to method
+|  	wrappers
+|  	[959d6bf1bdcd]
+|
+o  	* experiments/postgres-events.sql,
+|  	lib/observability/collector/timescale.rb:
+|  	Add more work on the timescale collector
+|  	[f0056a02fa49]
+|
+2019-07-18  Michael Granger  <ged@FaerieMUD.org>
+
+o  	* experiments/singleton-prepend.rb, experiments/thread-closures.rb,
+|  	lib/observability.rb, lib/observability/collector.rb,
+|  	lib/observability/collector/timescale.rb,
+|  	lib/observability/observer.rb, lib/observability/observer_hooks.rb,
+|  	lib/observability/sender/logger.rb,
+|  	spec/observability/event_spec.rb,
+|  	spec/observability/observer_spec.rb, spec/observability_spec.rb:
+|  	Start prototyping the collector, add observers for class methods
+|  	[21f0ed971661]
+|
+o  	* .gems, .hgignore, DevNotes.md, lib/observability.rb,
+|  	lib/observability/event.rb, lib/observability/observer.rb,
+|  	lib/observability/observer_hooks.rb, lib/observability/sender.rb,
+|  	lib/observability/sender/logger.rb, observability.gemspec,
+|  	spec/observability/observer_spec.rb,
+|  	spec/observability/sender/logger_spec.rb,
+|  	spec/observability/sender_spec.rb:
+|  	Add a bit more work on event and senders
+|  	[4413e7b15a86]
+|
+2019-07-17  Michael Granger  <ged@FaerieMUD.org>
+
+o  	* .gems, DevNotes.md, lib/observability.rb,
+|  	lib/observability/event.rb, lib/observability/observer.rb,
+|  	lib/observability/observer_hooks.rb,
+|  	spec/observability/event_spec.rb,
+|  	spec/observability/observer_hooks_spec.rb,
+|  	spec/observability/observer_spec.rb, spec/observability_spec.rb,
+|  	spec/spec_helper.rb:
+|  	Add base observation system
+|  	[f1670098182b]
+|
+2019-07-16  Michael Granger  <ged@FaerieMUD.org>
+
+o  	* lib/observability.rb, lib/observability/event.rb,
+|  	lib/observability/observer.rb, lib/observability/sender/testing.rb,
+|  	spec/observability/observer_spec.rb, spec/spec_helper.rb:
+|  	Checkpoint of more spec work
+|  	[bb75996b36b1]
+|
+2019-07-15  Michael Granger  <ged@FaerieMUD.org>
+
+o  	* .gems, Rakefile, lib/observability.rb,
+|  	lib/observability/observer.rb, lib/observability/observer_hooks.rb,
+|  	spec/observability_spec.rb, spec/spec_helper.rb:
+|  	Refine observer->sender and start working on specs
+|  	[afada6860d6b]
+|
+o  	* Rakefile, lib/observability.rb, lib/observability/event.rb,
+|  	lib/observability/sender.rb, lib/observability/sender/null.rb:
+|  	Add the base and default sender classes
+|  	[cb5a39a73863]
+|
+2019-07-13  Michael Granger  <ged@FaerieMUD.org>
+
+o  	* Rakefile, lib/observability.rb, lib/observability/event.rb,
+|  	lib/observability/observer.rb, lib/observability/observer_hooks.rb,
+|  	observability.gemspec:
+|  	Add more spike work
+|  	[9ac3e587898e]
+|
+2019-07-12  Michael Granger  <ged@FaerieMUD.org>
+
+o  	* experiments/superstar.rb, lib/observability.rb:
+|  	Flesh out more of the spike implementation
+|  	[50b6c0061b4a]
+|
+2019-07-11  Michael Granger  <ged@FaerieMUD.org>
+
+o  	* .gems, Rakefile, experiments/actors.rb, lib/observability.rb,
+|  	lib/observability/observer.rb, observability.gemspec:
+|  	Checkpoint of more spike code
+|  	[0b903d4090c9]
+|
+2019-07-09  Michael Granger  <ged@FaerieMUD.org>
+
+o  	* .gems, DevNotes.md, experiments/actors.rb, lib/observability.rb,
+|  	lib/observability/observer.rb, spec/observability_spec.rb,
+|  	spec/spec_helper.rb:
+|  	Checkpoint of notes and experiments
+|  	[0a1d6c04a68d]
+|
+2019-07-05  Michael Granger  <ged@FaerieMUD.org>
+
+o  	* DevNotes.md, examples/basic-usage.rb, lib/observability.rb:
+|  	Add notes and start of first example
+|  	[9851c06beeb7]
+|
+o  	* .document, .editorconfig, .gems, .hgignore, .pryrc, .rdoc_options,
+   	.ruby-gemset, .ruby-version, .simplecov, Gemfile, History.md,
+   	LICENSE.txt, Manifest.txt, README.md, Rakefile, certs/ged.pem,
+   	lib/observability.rb, spec/observability_spec.rb,
+   	spec/spec_helper.rb:
+   	Initial commit
+   	[2f9f3818be0c]

data/DevNotes.md

@@ -0,0 +1,103 @@
+# Developer Notes
+
+Characteristics of an observation:
+
+* Should it be timed?
+* What kinds of event models to support:
+  - Process: PID, PPID, PGID, rusage stuff
+  - Thread: Thread ID, PID, thread group ID, name, priority, thread_variables
+  - Net: Local IP, Peer IP, socktype, family, protocol, flags
+
+
+## Implementation
+
+Classes which are Observable will:
+
+* Create an "observer" Module that contains all of its observation hooks
+* Prepend the module onto the Observable class and its subclasses
+* Expose an instance method #observer that returns the module
+* Keep pending events and adds data to the innermost one via #observer.add
+
+The Observability module will:
+
+* Allow configuration of a sink to send events to
+* No-op if no sink is configured
+* Provide pluggable abstractions to:
+  - Provide the formatting of events
+  - Provide the transport to the event store
+
+
+## Possible Datapoints
+
+* Cardinality
+  - UUIDs
+  - db raw queries
+  - normalized queries
+  - comments
+  - PID/PPID
+  - app ID
+  - device ID
+  - HTTP headers
+  - build ID
+  - IP:port
+  - userid
+* Context
+  - 
+* Structured data
+* Tracing+events
+
+ 
+## Event Naming Convention
+
+Event names are dot-separated namespaces for events. You can use any convention that makes sense for your organization, but we suggest a convention like:
+
+    <namespace>.<system>.<verb>[.<detail>]+
+
+For a class named `FM::Manager::Players`, an event sent for a method called
+`connect_player` might be:
+
+    fm.manager.players.connect_player
+
+And an exception raised from that method call might generate:
+
+    fm.manager.players.connect_player.exception
+
+For the Sequel database toolkit, establishing a PostgreSQL connection might generate an event named:
+
+    sequel.adapters.postgres.connect
+
+and if the connection subsequently failed, it might be followed with:
+
+    sequel.adapters.postgres.connect.failure
+
+
+## Event "Models"
+
+If you pass a `:model` option when creating an event, you can pre-populate an event with a bunch of useful datapoints. Some ideas for models:
+
+
+`process`
+: Adds information about the current process: e.g., `getrusage(2)` deltas, `pid`, `pgid`, `ppid`, `uid`, `euid`, `egid`, `utime`, `stime`, `cutime`, `cstime`
+: The context should be a method that is called when a process starts and which returns when the process is going to shut down or has been shut down.
+
+`loop`
+: Adds information about the execution of a loop: e.g., `getrusage(2)` deltas, GC stats, etc.
+: It should be created immediately inside the loop to be measured.
+
+`thread`
+: Adds information about the execution of a thread: e.g., thread ID, name, priority, etc.
+
+
+
+## Unresolved Issues
+
+- How should events sent from class methods be distinguished from instance methods of the same name?
+- The object ID used for the event name for events sent from wrapped methods of singleton methods of anonymous classes is that of the singleton class, not the anonymous class itself. Given how infrequent this will be the case I'm not sure it's worth solving.
+- Is it worth the additional cost of duping arguments to callbacks of wrapper methods to ensure they don't accidentally modify them?
+- Make observing with a null sender be more no-opish.
+
+
+
+
+
+

data/History.md

@@ -0,0 +1,4 @@
+## v0.1.0 [2019-07-23] Michael Granger <ged@FaerieMUD.org>
+
+Initial release.
+

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2019 Michael Granger
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Manifest.txt

@@ -0,0 +1,31 @@
+.document
+.rdoc_options
+.simplecov
+ChangeLog
+DevNotes.md
+History.md
+LICENSE.txt
+Manifest.txt
+README.md
+Rakefile
+bin/observability-collector
+examples/basic-usage.rb
+lib/observability.rb
+lib/observability/collector.rb
+lib/observability/collector/timescale.rb
+lib/observability/event.rb
+lib/observability/observer.rb
+lib/observability/observer_hooks.rb
+lib/observability/sender.rb
+lib/observability/sender/logger.rb
+lib/observability/sender/null.rb
+lib/observability/sender/testing.rb
+lib/observability/sender/udp.rb
+spec/observability/event_spec.rb
+spec/observability/observer_hooks_spec.rb
+spec/observability/observer_spec.rb
+spec/observability/sender/logger_spec.rb
+spec/observability/sender/udp_spec.rb
+spec/observability/sender_spec.rb
+spec/observability_spec.rb
+spec/spec_helper.rb

data/README.md

@@ -0,0 +1,93 @@
+# Observability
+
+code
+: http://bitbucket.org/ged/observability
+
+github
+: https://github.com/ged/observability
+
+docs
+: http://deveiate.org/code/observability
+
+
+## Description
+
+Observability is a toolkit for instrumenting code to make it more observable,
+following the principle of Observability-Oriented Design as expressed by Charity
+Majors (@mipsytipsy).
+
+Its goals are [stolen from https://charity.wtf/2019/02/05/logs-vs-structured-events/]:
+
+* Emit a rich record from the perspective of a single action as the code is
+  executing.
+* Emit a single event per action per system that it occurs in. Write it out just
+  before the action completes or errors.
+* Bypass local disk entirely, write to a remote service.
+* Sample if needed for cost or resource constraints. Practice dynamic sampling.
+* Treat this like operational data, not transactional data. Be profligate and
+  disposable.
+* Feed this data into a columnar store or honeycomb or similar
+* Now use it every day. Not just as a last resort. Get knee deep in production
+  every single day. Explore. Ask and answer rich questions about your systems,
+  system quality, system behavior, outliers, error conditions, etc. You will be
+  absolutely amazed how useful it is … and appalled by what you turn up. 🙂
+
+
+## Prerequisites
+
+* Ruby 2.6
+
+
+## Installation
+
+    $ gem install observability
+
+
+## Contributing
+
+You can check out the current development source with Mercurial via its
+[project page][bitbucket]. Or if you prefer Git, via 
+[its Github mirror][github].
+
+After checking out the source, run:
+
+    $ rake newb
+
+This task will install any missing dependencies, run the tests/specs,
+and generate the API documentation.
+
+
+## License
+
+Copyright (c) 2019, Michael Granger
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice,
+  this list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the author/s, nor the names of the project's
+  contributors may be used to endorse or promote products derived from this
+  software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+[bitbucket]: http://bitbucket.org/ged/observability
+[github]: https://github.com/ged/observability
+