servicy 0.0.3

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: caf0064f89f13aec57352cff7a92d144db60444c
+  data.tar.gz: 7b95d789be54127997ed6ea51d709fd18910efe8
+SHA512:
+  metadata.gz: 0b3af8c73e8b8a5d2a61eef42c89dbef44b528a199e38436ff4ce1a54613d776a4f95fab796c73a736d9321b20917949d891b313ab66ab2cfcf86c011e3a729f
+  data.tar.gz: b4d288ad7756cd24814b5d57de1a15b8f50b1497fde40dc662ec1811d42127746337875afc3a2d6bb3da5417853729c4ff933710c81a84ce0cd1cc070188b1a1

data/README.md

@@ -0,0 +1,257 @@
+Servicy
+=======
+
+A service registration and discovery platform in Ruby for SOA.
+
+The goal of Servicy is to provide a simple daemon which allows services in an
+SOA application configuration to register themselves by name and/or API, and
+allow service consumers to find instances of services to use, and be relatively
+sure of up-time, routing, etc.
+
+Servicy could possibly used as a router as well, although that may be out of
+scope we will see.
+
+General architecture ideas
+==========================
+
+Servicy would be broken into a few pieces:
+
+1. A daemon which can handle registration and query requests
+2. An in-memory database of services to facilitate queries
+3. A persistent cache of services to allow for fast restarts
+4. (possibly) A service/API router
+5. Reporting service to report on service health and usage.
+6. Client library for service discovery and connection
+7. Service library for registration, API discovery and building
+
+One big issue that will have to be resolved is version dependencies. If there
+are two versions of a service, each with a different public API, then Servicy
+should know about the differences and send back to a service requester the
+correct version based on their requirements. For this, Servicy should have a
+way of discovering and/or services defining their API to Servicy.
+
+Each service should also be able to define multiple instances, and Servicy
+should have multiple strategies for choosing instances of a given service. For
+example, round-robin, load-based (would require machine-level meta-data to be
+reported by the services), even requests, etc. To start with, a simple
+round-robin scheme will likely be the only implementation. Furthermore, Servicy
+should have knowledge of if a service is up or down, and automatically remove
+from its list of available providers, or add it back in as needed.
+
+Also, Servicy should maintain statistics on usage for services and be able to
+report that information back to a user. Stats should include number of
+discoveries, cache hits/misses, service up-times, provider failures, and, if it
+works as a router, request latency. In the event that I end up _not_ making it
+a router (seems likely at this point), it should still report latency for
+service discovery and query operations that require back-and-forth with the
+service.
+
+Protocol ideas
+==============
+
+Servicy is protocol agnostic, and the exact transport mechanism used to
+communicate between client and server doesn't matter. The data is sent as JSON
+underneath it all (for now; I may even change things to be format agnostic in
+the future). You can see examples of the JSON by looking in the
+Servicy::Transport::Message class.
+
+Service registration
+--------------------
+
+Service registration allows for a service to declare that they exist, how to
+reach them, and what functionality they provide. Services are named using a
+Java-style, inverted domain name. For example:
+
+```
+com.mycompany.users.authentication
+```
+
+might be the name of a user authentication service provided internally. This
+allows for services to be broken up by what other services they interact with,
+as well as be easy to read for users browsing the registry.
+
+Beyond the name, a service must provide some other information for its
+registry. This includes:
+
+1. Host to connect to
+2. Port on the host to connect to
+3. Protocol used to communicate (HTTP, HTTPS, Thrift, etc.)
+4. Version number
+5. Heartbeat port
+
+The host can be either an IP address or a fully-qualified hostname. In either
+case it should be route-able in the context of the Servicy host. In other
+words, if Servicy cannot connect with it, it won't add it to the registry. The
+port should be an integer between 1 and 65535. The protocol can be anything,
+however some standards are:
+
+* HTTP - _Must_ use un-encrypted HTTP
+* HTTPS - _Must_ use TLS over HTTP
+* HTTP/S - Can use either HTTP or HTTPS
+* Thrift
+* TCP - Direct, telnet-like communication
+* UDP - Direct connection, but don't expect replies
+
+The protocol says nothing about what information needs to be sent or received
+by a client consuming the service. It is assumed that if the client is looking
+for the service, it knows how to use it once it is found, and will only use the
+protocol information to select between possible options provided by the
+consuming library. Protocols can also be provided as a list of possible
+options, ordered with most preferred first. At some point in the future, there
+may be a mechanism by which a service can describe itself in a more useful way,
+and client libraries can use that information to dynamically build an internal
+API.
+
+The version number must follow the format:
+
+```
+major.minor.revision-patch
+```
+
+The "-patch" part can be omitted. For example:
+
+```
+1.0.2-p143
+```
+
+The heartbeat port is a port that Servicy can connect to to ensure that the
+service is still functioning. In many cases this will be the same port as the
+primary connection port; Servicy does not attempt to use the heartbeat
+connection for anything. It simply connects, and if successful, disconnects
+again. For this reason, your services should be resilliant to this type of
+traffic.
+
+Along with the required information already listed, a service can provide some
+optional information about the API that they provide. This information comes
+with some pieces of information for each API endpoint or action provided. The
+information is:
+
+1. Endpoint identifier
+2. Parameter information
+3. Return information
+4. (optional) Human-readable documentation.
+
+The endpoint identifier can be anything, and is just a string. For example, in
+an HTTP/S RESTful API, it may be something like:
+
+```
+/api/v1/user/login
+```
+
+The parameter information is information about each parameter that is expected
+by this endpoint. This data is structured, and requires information about name,
+data type, and weather or not the parameter is required.
+
+The return information is simply a description of the data that should be
+expected to return.
+
+The documentation is an arbitrary-length text string which can be used as
+documentation by a developer developing against the API.
+
+Multiple service providers can provide the same service. In that case, they are
+simply added to a pool of possible providers that is selected from using some
+load-balancing strategy such as round-robin or random selection.
+
+Any service provider that has registered will periodically get ICMP pinged by
+Servicy. If a response does not come back, it will be removed from the pool of
+providers. If a response does come back, Servicy will then attempt to connect
+to the heartbeat port to determine if the service is still functioning on the
+box. If it is not, the provider will be removed from the pool, otherwise it
+will be left in. This two-step process is to facilitate better logging and
+debugging of issues.
+
+Service de-registration
+-----------------------
+
+A service can remove itself from the pool voluntarily be simply sending the
+name, version, host, and port of the service to be removed. However, this is
+not strictly needed as the periodic heartbeat check will notice if a service
+goes away. These checks happen every second.
+
+A note on security
+++++++++++++++++++
+
+For the time-being, it is assumed that you will have Servicy and all the
+services it interacts with installed on internal, secured infrastructure. As
+such, Servicy takes a very trusting policy. However, that means that someone
+with malintent and access to your network could de-register services, or
+register a bunch of fake ones. Future versions of Servicy may attempt to
+resolve this issue. We'll see...
+
+Service discovery
+-----------------
+
+Once services are registered, a client needs to be able to discover them.
+Generally, they will discover a service and hold on to that information for the
+lifetime of their use, only requesting a new handle should the old one be lost.
+However, there is no reason why they couldn't request a new one every time.
+
+*NOTE* In the future, it may be possible for Servicy to act as a router by
+simply returning connection information for itself, and doing blind
+pass-through with some strategy to the other providers. In that case, the
+policy that a client uses to cache or not cache their connection information
+may have to change.
+
+A service consumer can query using one or both of two different dimensions:
+
+1. API name
+2. API functionality
+
+When using the name dimension, you can also query by version using either an
+exact match, a list of preferences, a range, a minimum, or a maximum.
+
+When querying using functionality, you query using one or more API method
+fingerprints. The fingerprints are in the following format:
+
+```
+method_name#arg-type,arg-type,...#return_type
+```
+
+where method_name is the name of the method you would like to call (or service
+end-point, or whatever. This is the same as the API endpoint in the service
+functionality registration information.), the `arg-type` pairs are the name of
+an argument (or an empty string in the case of un-named arguments), followed by
+a dash, followed by the argument type, and the return_type being the data-type
+for the return value.
+
+For example, a query could look something like this:
+
+```
+user_create#username-string,password-string#User
+```
+
+Behind the scenes, Servicy will use this information to find all possible
+services that can fulfill this request. This will include any service that can
+perform this request but that may have more optional parameters. Type
+information can also be provided as a list of types, in the case of API's that
+can handle multiple types of information:
+
+```
+user_create#username-string,password-string|signed_string#User
+```
+
+The possible other types that the data can be are separated by a pipe, '|'.
+
+Finally, the service consumer can provide a boolean flag indicating weather or
+not they wish to receive API information about the returned providers, or if
+they simply want connection information. If the flag is set to true, they will
+get full API documentation and definitions along with connection information.
+
+The query system will either provide a successful "error"-code, along with a
+list of one or more service providers, a "not-found" error-code indicating that
+the query was valid, but that there are no available services which meet the
+requirements, or an "error" error-code that indicates that something was wrong
+with the provided query. In the last two queries, additional error information
+may be provided, but should be considered for human-use only; ie, showing an
+error message to a user.
+
+TODO
+====
+
+Things left to do:
+
+1. In the command-line tool, there is some kind of bug where the registration
+   message never returns back to the client. This causes registration to hang
+   forever...
+1. There are a number of just kind of ugly-code things around that I should sort out.
+1. Probably some other things that I'm forgetting.

data/Servicy.gemspec

@@ -0,0 +1,47 @@
+# Generated by jeweler
+# DO NOT EDIT THIS FILE DIRECTLY
+# Instead, edit Jeweler::Tasks in Rakefile, and run 'rake gemspec'
+# -*- encoding: utf-8 -*-
+# stub: servicy 0.0.3 ruby lib
+
+Gem::Specification.new do |s|
+  s.name = "servicy"
+  s.version = "0.0.3"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.require_paths = ["lib"]
+  s.authors = ["Thomas Luce"]
+  s.date = "2014-08-11"
+  s.description = "A service registration and discovery framework, as a server and client library."
+  s.email = "thomas.luce@gmail.com"
+  s.executables = ["servicy"]
+  s.extra_rdoc_files = [
+    "README.md"
+  ]
+  s.files = [
+    "README.md",
+    "Servicy.gemspec",
+    "VERSION",
+    "bin/servicy",
+    "lib/api.rb",
+    "lib/client.rb",
+    "lib/hash.rb",
+    "lib/load_balancer.rb",
+    "lib/load_balancer/random.rb",
+    "lib/load_balancer/round_robin.rb",
+    "lib/server.rb",
+    "lib/server/server.rb",
+    "lib/server/service_searcher.rb",
+    "lib/service.rb",
+    "lib/servicy.rb",
+    "lib/transport/in_memory_transport.rb",
+    "lib/transport/messages.rb",
+    "lib/transport/tcp_transport.rb",
+    "lib/transports.rb",
+    "test.rb"
+  ]
+  s.homepage = "https://github.com/thomasluce/servicy"
+  s.rubygems_version = "2.2.2"
+  s.summary = "A service registration and discovery framework"
+end
+

data/VERSION

@@ -0,0 +1 @@
+0.0.3

data/bin/servicy

@@ -0,0 +1,305 @@
+#!/usr/bin/env ruby
+
+require 'optparse'
+require 'ostruct'
+require 'rubygems'
+require 'servicy'
+require 'transport/tcp_transport'
+require 'pp'
+require 'fileutils'
+
+VERSION = File.read(File.expand_path(File.join(File.dirname(__FILE__), '..', 'VERSION')))
+
+# The following couple of functions are taking from the daemons gem. I don't
+# need all it's functionality, and it's configuration options don't give me
+# what I want. Given that I only need very simple daemonizing options here,
+# this will be fine.
+def safefork
+  tryagain = true
+  while tryagain
+    tryagain = false
+    begin
+      if pid = fork
+        return pid
+      else
+        return false
+      end
+    rescue Errno::EWOULDBLOCK
+      sleep 5
+      tryagain = true
+    end
+  end
+end
+
+def redirect_io(logfile_name)
+  begin; STDIN.reopen "/dev/null"; rescue ::Exception; end
+
+  if logfile_name
+    begin
+      STDOUT.reopen logfile_name, "a"
+      File.chmod(0644, logfile_name)
+      STDOUT.sync = true
+    rescue ::Exception
+      begin; STDOUT.reopen "/dev/null"; rescue ::Exception; end
+    end
+  else
+    begin; STDOUT.reopen "/dev/null"; rescue ::Exception; end
+  end
+
+  begin; STDERR.reopen STDOUT; rescue ::Exception; end
+  STDERR.sync = true
+end
+
+def make_daemon(logfile_name=nil, pidfile_name=nil)
+  # Fork and exit from the parent
+  safefork and exit
+
+  # Detach from the controlling terminal
+  unless sess_id = Process.setsid
+    raise 'cannot detach from controlling terminal'
+  end
+
+  # Prevent the possibility of acquiring a controlling terminal
+  trap 'SIGHUP', 'IGNORE'
+  exit if pid = safefork
+
+  # Release old working directory
+  Dir.chdir "/"
+
+  # Make stdout and stderr go to our log if defined. /dev/null otherwise.
+  redirect_io(logfile_name)
+
+  # Split rand streams between spawning and daemonized process
+  srand
+
+  # Write a pidfile
+  File.open(pidfile_name, 'w') { |f| f.puts Process.pid }
+
+  # Set the service name so that ps makes sense.
+  $0 = 'servicy'
+
+  return sess_id
+end
+
+class ServicyOptionParser
+  def self.parse(command, command_command, args)
+    options = OpenStruct.new
+    options.port            = 8000
+    options.daemonize       = false
+    options.config_file     = File.expand_path("~/.servicy.conf")
+    options.pid_file        = File.expand_path("./servicy.pid")
+    options.log_file        = File.expand_path('./servicy.log')
+    options.service_version = "1.0.0"
+    options.protocol        = "HTTP/S"
+    options.server_host     = "localhost"
+    options.command         = %w(client server).include?(command) ? command : nil
+    options.command_command = %w(start stop status info register unregister search).include?(command_command) ? command_command : nil
+
+    OptionParser.new do |opts|
+      opts.banner = "Usage: servicy [command] [command command] [command options]"
+      opts.separator "Where command is one of: server, client"
+      opts.separator "And command-command is a command to send to either the client or the server."
+      opts.separator ""
+
+      opts.separator "Server commands:"
+      opts.separator "     start  - Start the server"
+      opts.separator "     stop   - Stop a running server"
+      opts.separator "     status - Get weather or not a server is running"
+      opts.separator "     info   - Get info about a running server"
+
+      opts.separator ""
+      opts.separator "Server command options:"
+
+      opts.on '-p', '--port PORT', Integer, "Set the port for the server to listen on. Defaults to #{options.port}" do |port|
+        options.port = port
+      end
+
+      opts.on '-d', '--daemon', "Daemonize after starting server. Defaults to #{options.daemonize}" do
+        options.daemonize = true
+      end
+
+      opts.on '-f', '--config CONFIG_FILE', String, "Set the location for the service config file. Defaults to #{options.config_file}" do |file|
+        options.config_file = File.expand_path(file)
+      end
+
+      opts.on '-i', '--pid PID_FILE', String, "Set the PID file location. Defaults to #{options.pid_file}" do |file|
+        options.pid_file = File.expand_path(file)
+      end
+
+      opts.on '-l', '--logfile LOG_FILE', String, "Location of log file. Defaults to #{options.log_file}" do |file|
+        options.log_file = File.expand_path(file)
+      end
+
+      opts.separator ""
+      opts.separator "Client commands:"
+      opts.separator "     register   - Register a new service"
+      opts.separator "     unregister - Un-register a registered service"
+      opts.separator "     search     - Search for a registered service"
+
+      opts.separator ""
+      opts.separator "Client command options:"
+
+      opts.on '-n', '--name NAME', String, "The name of the service" do |name|
+        options.service_name = name
+      end
+
+      opts.on '-h', '--host HOST', String, "The hostname or IP-address of the service" do |host|
+        options.service_host = host
+      end
+
+      opts.on '-H', '--server-host HOST', String, "The host where the service registrar can be found. Defaults to #{options.server_host}" do |host|
+        options.server_host = host
+      end
+
+      opts.on '-s', '--service-port PORT', Integer, "The port the service is on" do |port|
+        options.service_port = port
+      end
+
+      opts.on '-b', '--heartbeat-port PORT', Integer, "The port that the services' heartbeat service lives. Defaults to the service port" do |port|
+        options.heartbeat_port = port
+      end
+
+      opts.on '-e', '--service-version VERSION', String, "The version of the service. Defaults to #{options.service_version}" do |version|
+        options.version = version
+      end
+
+      opts.on '-r', '--protocol PROTO', String, "The protocol of the service. Defaults to #{options.protocol}" do |protocol|
+        options.protocol = protocol
+      end
+
+      opts.on '-a', '--api API', String, "API description for the service. See documentation for shorthand help." do |api|
+        options.api = api
+      end
+
+      opts.separator ""
+      opts.separator "Common options:"
+
+      opts.on "--help", "Show this message and exit" do
+        puts opts
+        exit
+      end
+
+      opts.on "-v", "--version", "Show current version and exit" do
+        puts VERSION
+        exit
+      end
+
+      opts.separator ""
+      opts.separator "Examples:"
+      opts.separator ""
+
+      opts.separator "Start a service discovery/registration server on port 3214"
+      opts.separator "  servicy server start -p 3214"
+      opts.separator ""
+
+      opts.separator "Register a service at the server on 192.168.0.1, port 1234"
+      opts.separator "  servicy client register -n com.foo.bar -p 1234 -H localhost -h localhost -s 1234"
+      opts.separator ""
+
+      opts.separator "Find a service"
+      opts.separator "  servicy client search -H 192.168.0.1 -n 'com.bar.foo'"
+      opts.separator ""
+
+      if !command || !command_command
+        puts opts
+        exit
+      end
+    end.parse!
+
+    return options
+  end
+end
+
+class Command
+  def initialize(options)
+    @options = options
+  end
+
+  def method_missing(name, *args, &block)
+    @options[name]
+  end
+end
+
+class ServerCommand < Command
+  def start
+    transport = Servicy::TCPTransport.new(port: port)
+    if File.exists?(config_file)
+      server = Servicy::Server.load(config_file)
+      server.transport = transport
+    else
+      server = Servicy::Server.new(transport, nil, STDOUT, config_file)
+    end
+
+    make_daemon(log_file, pid_file) if daemonize
+    server.go!
+  end
+
+  def stop
+    # TODO: At some point in the future, this should send a shutdown command to
+    # give the server time to do its thing gracefully.
+    raise "Pid not found" if pid == 0 || !pid
+    Process.kill('TERM', pid)
+    FileUtils.rm(pid_file)
+  end
+
+  def info
+    client = Servicy::Client.new(Servicy::TCPTransport.new(port: port))
+    pp client.server_stats.struct
+  end
+
+  def status
+    Process.getpgid(pid)
+    puts "Server is running"
+  rescue
+    puts "Server is not running"
+  end
+
+  private
+
+  def pid
+    File.read(pid_file).strip.to_i
+  end
+end
+
+class ClientCommand < Command
+  def register
+    client.register_service({name: service_name,
+                            host: service_host,
+                            port: service_port,
+                            heatbeat_port: heartbeat_port,
+                            version: version,
+                            protocol: protocol
+    })
+
+    # TODO: api parsing and registration. I want to get the search done for
+    # that first.
+  end
+
+  def unregister
+  end
+
+  def search
+    puts client.find_service(name: service_name,
+                             host: service_host,
+                             port: service_port,
+                             heartbeat_port: heartbeat_port,
+                             version: version,
+                             protocol: protocol
+                            ).as_json.to_json
+  end
+
+  private
+
+  def client
+    Servicy::Client.new(Servicy::TCPTransport.new(port: port))
+  end
+end
+
+if ARGV.first.to_s.start_with? "-"
+  ServicyOptionParser.parse(nil, nil, ARGV)
+  exit # Help or version
+else
+  options = ServicyOptionParser.parse(ARGV.shift, ARGV.shift, ARGV)
+  klass = options.command == 'server' ? ServerCommand : ClientCommand
+  klass.new(options).send(options.command_command)
+end