com.adrenak.univoice 3.0.0 → 4.0.0

package/README.md

@@ -1,5 +1,3 @@
-Note: Inbuilt implementations and samples have been removed from this repository. They'll be added to separate repositories soon.
-
 # UniVoice
 UniVoice is a voice chat/VoIP solution for Unity.
   
@@ -9,73 +7,53 @@ Some features of UniVoice:
 - ⚙ Peer specific settings. Don't want to listen to a peer? Mute them. Don't want someone listening to you? Mute yourself against them.
  
 - 🎨 Customize your audio input, output and networking layer. 
-  * 🎤 __Configurable Audio Input__: Decide what the input of your outgoing audio is. Let it be from [Unity's Microphone](https://docs.unity3d.com/ScriptReference/Microphone.html) class, or a live streaming audio, or an MP4 file on the disk.
+  * 🎤 __Configurable Audio Input__: UniVoice is audio input agnostic. It supports mic audio input out of the box and you can change the source of outgoing audio by implementing the `IAudioInput` interrace.  
     
-  * 🔊 __Configurable Audio Output__:  Decide where the incoming peer audio goes. Let the output of incoming audio be [Unity AudioSource](https://docs.unity3d.com/ScriptReference/AudioSource.html) to play the audio in-game, or write it into an MP4 on the disk, or stream it to an online service.
+  * 🔊 __Configurable Audio Output__:  UniVoice is audio output agnostic. Out of the box is supports playing peer audio using Unity AudioSource. You can divert incoming audio to anywhere you want by implementing the `IAudioOutput` interface.  
 
-  * 🌐 __Configurable Network__: Want to use UniVoice in a WLAN project using [Telepathy?](https://github.com/vis2k/Telepathy) Just adapt its API for UniVoice with a simple the `IChatroomNetwork` interface. Using your own backend for multiplayer? Create and expose your audio API and write a UniVoice implementation, again with the same interface.
-  
-# Docs
-Manuals and sample projects are not available yet. For the API reference, please visit http://www.vatsalambastha.com/univoice
-  
-# Usage
-## Creating a chatroom agent
-- To be able to host and join voice chatrooms, you need a `ChatroomAgent` instance.
+  * 🌐 __Configurable Network__: UniVoice is network agnostic and supports Mirror out of the box. You can implement the `IAudioClient` and `IAudioServer` interfaces using the networking plugin of your choice to make it compatible with it.
   
+## Installation
+⚠️ [OpenUPM](https://openupm.com/packages/com.adrenak.univoice/?subPage=versions) may not have up to date releases. Install using NPM registry instead 👇
+
+Ensure you have the NPM registry in the `packages.json` file of your Unity project with the following scopes:
 ```
-var agent = new ChatroomAgent(IChatroomNetwork network, IAudioInput audioInput, IAudioOutput audioOutput);
+"scopedRegistries": [
+    {
+        "name": "npmjs",
+        "url": "https://registry.npmjs.org",
+        "scopes": [
+            "com.npmjs",
+            "com.adrenak.univoice",
+            "com.adrenak.brw",
+            "com.adrenak.unimic",
+            "com.adrenak.unityopus"
+        ]
+    }
+}
 ```
 
-## Hosting and joining chatrooms
-
-Every peer in the chatroom is assigned an ID by the host. And every peer has a peer list, representing the other peers in the chatroom.
+## Docs
+Am API reference is available: http://www.vatsalambastha.com/univoice
 
-- To get your ID  
-`agent.Network.OwnID;`
+## Samples
+This repository contains a sample scene for the Mirror network, which is the best place to see how UniVoice can be integrated into your project.  
   
-- To get a list of the other peers in the chatroom, use this:  
-`agent.Network.PeersIDs`
-
-`agent.Network` also provides methods to host or join a chatroom. Here is how you use them:
+To try the sample, import Mirror and add the `UNIVOICE_MIRROR_NETWORK` compilation symbol to your project.
   
-```
-// Host a chatroom using a name
-agent.Network.HostChatroom(optional_data);
+## Dependencies
+[com.adrenak.brw](https://www.github.com/adrenak/brw)`@1.0.1` for reading and writing messages for communication. See `MirrorServer.cs` and `MirrorClient.cs` where they're used.  
 
-// Join an existing chatroom using a name
-agent.Network.JoinChatroom(optional_data);
+[com.adrenak.unimic](https://www.github.com/adrenak/unimic)`@3.2.1` for easily capturing audio from any connected mic devices. See `UniMicInput.cs` for usage.  
 
-// Leave the chatroom, if connected to one
-agent.Network.LeaveChatroom(optional_data);
+[com.adrenak.unityopus](https://www.github.com/adrenak/unityopus)`@1.0.0` for Opus encoding and decoding. See `OpusFilter.cs` for usage
 
-// Closes a chatroom, if is hosting one
-agent.Network.CloseChatroom(optional_data);
-
-```
-## Muting Audio
-To mute everyone in the chatroom, use `agent.MuteOthers = true;` or set it to `false` to unmute them all.  
-  
-To mute yourself use `agent.MuteSelf = true;` or set it to `false` to unmute yourself. This will stop sending your audio to all the peers in the chatroom.
-
-For muting a specific peer, first get the peers settings object using this:  
-```
-agent.PeerSettings[id].muteThem = true; // where id belongs to the peer in question
-```
-  
-If you want to mute yourself towards a specific peer, use this:
-`agent.PeerSettings[id].muteSelf = true; // where id belongs to the peer in question`
-  
-## Events
-`agent.Network` provides several network related events. Refer to the [API reference](http://www.vatsalambastha.com/univoice/api/Adrenak.UniVoice.ChatroomAgent.html) for them.
-
-# License and Support
+## License and Support
 This project is under the [MIT license](https://github.com/adrenak/univoice/blob/master/LICENSE).
 
-Updates and maintenance are not guaranteed and the project is maintained by the original developer in his free time. Community contributions are welcome.
-  
-__Commercial consultation and development can be arranged__ but is subject to schedule and availability.  
+Community contributions are welcome.
   
-# Contact
+## Contact
 The developer can be reached at the following links:
   
 [Website](http://www.vatsalambastha.com)  

package/Runtime/Adrenak.UniVoice.Runtime.asmdef

@@ -1,3 +1,19 @@
-{
-	"name": "Adrenak.UniVoice.Runtime"
-}
+{
+    "name": "Adrenak.UniVoice.Runtime",
+    "references": [
+        "GUID:f87ecb857e752164ab814a3de8eb0262",
+        "GUID:1f776cd02c03a7b4280b6b649d7758e2",
+        "GUID:30817c1a0e6d646d99c048fc403f5979",
+        "GUID:725ee7191c021de4dbf9269590ded755",
+        "GUID:34d15e3fb5fa4b541b5a93a6dc182cc5"
+    ],
+    "includePlatforms": [],
+    "excludePlatforms": [],
+    "allowUnsafeCode": false,
+    "overrideReferences": false,
+    "precompiledReferences": [],
+    "autoReferenced": true,
+    "defineConstraints": [],
+    "versionDefines": [],
+    "noEngineReferences": false
+}

package/Runtime/ClientSession.cs

@@ -0,0 +1,137 @@
+using System;
+using System.Collections.Generic;
+
+using UnityEngine;
+
+namespace Adrenak.UniVoice {
+    /// <summary>
+    /// Handles a client session. 
+    /// Requires an implementation of <see cref="IAudioClient{T}"/>, <see cref="IAudioInput"/> and <see cref="IAudioOutputFactory"/> each.
+    /// Allows adding input and output filters and handles their execution.
+    /// </summary>
+    /// <typeparam name="T"></typeparam>
+    public class ClientSession<T> : IDisposable {
+        /// <summary>
+        /// The <see cref="IAudioOutput"/> instances of each peer in the session
+        /// </summary>
+        public Dictionary<T, IAudioOutput> PeerOutputs { get; private set; } = new Dictionary<T, IAudioOutput>();
+
+        /// <summary>
+        /// The input <see cref="IAudioFilter"/> that will be applied to the outgoing audio for all the peers.
+        /// Note that filters are executed in the order they are present in this list
+        /// </summary>
+        public List<IAudioFilter> InputFilters { get; set; } = new List<IAudioFilter>();
+
+        /// <summary>
+        /// The output <see cref="IAudioFilter"/> that will be applied to the incoming audio for all the peers.
+        /// Note that filters are executed in the order they are present in this list.
+        /// </summary>
+        public List<IAudioFilter> OutputFilters { get; set; } = new List<IAudioFilter>();
+
+        public ClientSession(IAudioClient<T> client, IAudioInput input, IAudioOutputFactory outputFactory) {
+            Client = client;
+            Input = input;
+            OutputFactory = outputFactory;
+        }
+
+        /// <summary>
+        /// The <see cref="IAudioClient{T}"/> that's used for networking
+        /// </summary>
+        IAudioClient<T> client;
+        public IAudioClient<T> Client {
+            get => client;
+            set {
+                if(client != null)
+                    client.Dispose();
+                client = value;
+
+                Client.OnLeft += () => {
+                    foreach (var output in PeerOutputs)
+                        output.Value.Dispose();
+                    PeerOutputs.Clear();
+                };
+
+                Client.OnPeerJoined += id => {
+                    try {
+                        var output = outputFactory.Create();
+                        PeerOutputs.Add(id, output);
+                    }
+                    catch (Exception e) {
+                        Debug.LogException(e);
+                    }
+                };
+
+                Client.OnPeerLeft += id => {
+                    if (!PeerOutputs.ContainsKey(id)) 
+                        return;
+
+                    PeerOutputs[id].Dispose();
+                    PeerOutputs.Remove(id);
+                };
+
+                client.OnReceivedPeerAudioFrame += (id, audioFrame) => {
+                    if (!PeerOutputs.ContainsKey(id))
+                        return;
+
+                    if (OutputFilters != null) {
+                        foreach (var filter in OutputFilters)
+                            audioFrame = filter.Run(audioFrame);
+                    }
+
+                    PeerOutputs[id].Feed(audioFrame);
+                };
+            }
+        }
+
+        IAudioInput input;
+        /// <summary>
+        /// The <see cref="IAudioInput"/> that's used for sourcing outgoing audio
+        /// </summary>
+        public IAudioInput Input {
+            get => input;
+            set {
+                if(input != null)
+                    input.Dispose();
+                input = value;
+                input.OnFrameReady += frame => {
+                    if (InputFilters != null) {
+                        foreach (var filter in InputFilters)
+                            frame = filter.Run(frame);
+                    }
+
+                    Client.SendAudioFrame(frame);
+                };
+            }
+        }
+
+        IAudioOutputFactory outputFactory;
+        /// <summary>
+        /// The <see cref="IAudioOutputFactory"/> that creates the <see cref="IAudioOutput"/> of peers
+        /// </summary>
+        public IAudioOutputFactory OutputFactory {
+            get => outputFactory;
+            set {
+                outputFactory = value;
+
+                foreach (var output in PeerOutputs)
+                    output.Value.Dispose();
+                PeerOutputs.Clear();
+
+                foreach (var id in Client.PeerIDs) {
+                    try {
+                        var output = outputFactory.Create();
+                        PeerOutputs.Add(id, output);
+                    }
+                    catch (Exception e) {
+                        Debug.LogException(e);
+                    }
+                }
+            }
+        }
+
+        public void Dispose() {
+            Client.Dispose();
+            Input.Dispose();
+        }
+    }
+}

package/Runtime/{Interfaces/IChatroomNetwork.cs.meta → ClientSession.cs.meta}

@@ -1,5 +1,5 @@
 fileFormatVersion: 2
-guid: 11ba8951ad10dc84a963ca3c32cda48a
+guid: 923d8509d2c05f2479d6af7331dabe58
 MonoImporter:
   externalObjects: {}
   serializedVersion: 2

package/Runtime/Common/Utils.cs

@@ -0,0 +1,55 @@
+using System.IO;
+using System.IO.Compression;
+using System.Net.Sockets;
+using System.Net;
+using System.Runtime.Serialization.Formatters.Binary;
+using System;
+using UnityEngine;
+
+namespace Adrenak.UniVoice {
+    public class Utils {
+        public class Bytes {
+            public static byte[] FloatsToBytes(float[] floats) {
+                int byteCount = sizeof(float) * floats.Length;
+                byte[] byteArray = new byte[byteCount];
+
+                Buffer.BlockCopy(floats, 0, byteArray, 0, byteCount);
+
+                return byteArray;
+            }
+
+            public static float[] BytesToFloats(byte[] bytes) {
+                int floatCount = bytes.Length / sizeof(float);
+                float[] floatArray = new float[floatCount];
+
+                Buffer.BlockCopy(bytes, 0, floatArray, 0, bytes.Length);
+
+                return floatArray;
+            }
+        }
+
+        public static class Audio {
+            static float[] audioF;
+            static float sumOfSquares;
+            public static float CalculateRMS(byte[] audio) {
+                audioF = Bytes.BytesToFloats(audio);
+
+                foreach(var x in audioF)
+                    sumOfSquares += x * x;
+                return Mathf.Sqrt(sumOfSquares / audioF.Length);
+            }
+        }
+
+        public static class Network {
+            public static string LocalIPv4Address {
+                get {
+                    using (Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, 0)) {
+                        socket.Connect("8.8.8.8", 65530);
+                        IPEndPoint endPoint = socket.LocalEndPoint as IPEndPoint;
+                        return endPoint.Address.ToString();
+                    }
+                }
+            }
+        }
+    }
+}

package/Runtime/{Types/ChatroomAgentMode.cs.meta → Common/Utils.cs.meta}

@@ -1,5 +1,5 @@
 fileFormatVersion: 2
-guid: 95ab24cb9f03b8c4c8f22f567259def4
+guid: a3530a416ca433448ab4c50826faefb4
 MonoImporter:
   externalObjects: {}
   serializedVersion: 2

package/Runtime/Common.meta

@@ -0,0 +1,8 @@
+fileFormatVersion: 2
+guid: 7e8f97d020fcd2740b34e8d69e4f882d
+folderAsset: yes
+DefaultImporter:
+  externalObjects: {}
+  userData: 
+  assetBundleName: 
+  assetBundleVariant: 

package/Runtime/Impl/Filters/GaussianAudioBlur.cs

@@ -0,0 +1,90 @@
+using System;
+
+using UnityEngine;
+
+namespace Adrenak.UniVoice.Filters {
+    /// <summary>
+    /// A filter that applies Gaussian blur over audio data to smoothen it.
+    /// This is somewhat effective in removing noise from the audio.
+    /// </summary>
+    public class GaussianAudioBlur : IAudioFilter {
+        readonly float sigma;
+        readonly int range;
+        byte[] lastInput;
+
+        public GaussianAudioBlur(float sigma = 2, int range = 2) {
+            this.sigma = sigma;
+            this.range = range;
+        }
+
+        public AudioFrame Run(AudioFrame frame) {
+            var input = frame.samples;
+            if (input == null || input.Length == 0) {
+                frame.samples = null;
+                return frame;
+            }
+
+            // If this is the first audio input we've received, we simply apply the gaussian filter
+            // and return the result.
+            if (lastInput == null) {
+                lastInput = input;
+                frame.samples = Utils.Bytes.FloatsToBytes(
+                    ApplyGaussianFilter(Utils.Bytes.BytesToFloats(input))
+                );
+                return frame;
+            }
+
+            // Else, if we've had some input before, we also consider the previously processed 
+            // audio. We make an array that has both the previous and the current input, smoothen 
+            // it, and then return the second half of the array. This reducing jittering by making
+            // the smoothing a little more seamless.
+            else {
+                // Create an all input, that also has the input from the last time this filter ran.
+                byte[] allInput = new byte[lastInput.Length + input.Length];
+                Buffer.BlockCopy(lastInput, 0, allInput, 0, lastInput.Length);
+                Buffer.BlockCopy(input, 0, allInput, lastInput.Length, input.Length);
+
+                // smoothen all input
+                byte[] allInputSmooth = Utils.Bytes.FloatsToBytes(
+                    ApplyGaussianFilter(Utils.Bytes.BytesToFloats(allInput))
+                );
+
+                // get the second half of the smoothened values
+                byte[] result = new byte[input.Length];
+                Buffer.BlockCopy(allInputSmooth, lastInput.Length, result, 0, input.Length);
+
+                lastInput = input;
+                frame.samples = result;
+                return frame;
+            }
+        }
+
+        float[] ApplyGaussianFilter(float[] inputArray) {
+            int length = inputArray.Length;
+            float[] smoothedArray = new float[length];
+
+            for (int i = 0; i < length; i++) {
+                float sum = 0.0f;
+                float weightSum = 0.0f;
+
+                for (int j = -range; j <= range; j++) {
+                    int index = i + j;
+                    if (index >= 0 && index < length) {
+                        float weight = Gaussian(j, sigma);
+                        sum += inputArray[index] * weight;
+                        weightSum += weight;
+                    }
+                }
+
+                smoothedArray[i] = sum / weightSum;
+            }
+
+            return smoothedArray;
+        }
+
+        float Gaussian(int x, float sigma) {
+            return (float)Mathf.Exp(-(x * x) / (2 * sigma * sigma))
+            / ((float)Mathf.Sqrt(2 * Mathf.PI) * sigma);
+        }
+    }
+}

package/Runtime/{ChatroomAgent.cs.meta → Impl/Filters/GaussianAudioBlur.cs.meta}

@@ -1,5 +1,5 @@
 fileFormatVersion: 2
-guid: 76706adfc97ef7d44823aa51d618267d
+guid: 5992899a0c52b09498191b477927699d
 MonoImporter:
   externalObjects: {}
   serializedVersion: 2

package/Runtime/Impl/Filters/OpusFilter.cs

@@ -0,0 +1,85 @@
+using Adrenak.UnityOpus;
+
+using System;
+
+/*
+ * Opus encoding and decoding are VERY important for any real world use of UniVoice as without
+ * encoding the size of audio data is much (over 10x) larger.
+ * For more info see https://www.github.com/adrenak/UnityOpus
+ */
+namespace Adrenak.UniVoice.Filters {
+    /// <summary>
+    /// A filter that encodes audio using Opus. Use this as an output filter
+    /// to reduce the size of outgoing client audio
+    /// </summary>
+    public class OpusEncodeFilter : IAudioFilter {
+        Encoder encoder;
+        byte[] outputBuffer;
+
+        public OpusEncodeFilter(Encoder encoder) {
+            this.encoder = encoder;
+        }
+
+        public AudioFrame Run(AudioFrame input) {
+            if(outputBuffer == null)
+                outputBuffer = new byte[input.samples.Length * 4];
+            else if(input.samples.Length != outputBuffer.Length * 4)
+                outputBuffer = new byte[input.samples.Length * 4];
+
+            int encodeResult = encoder.Encode(Utils.Bytes.BytesToFloats(input.samples), outputBuffer);
+            if (encodeResult > 0) {
+                byte[] encodedBytes = new byte[encodeResult];
+                Array.Copy(outputBuffer, encodedBytes, encodedBytes.Length);
+                return new AudioFrame {
+                    timestamp = 0,
+                    frequency = input.frequency,
+                    channelCount = input.channelCount,
+                    samples = encodedBytes
+                };
+            }
+            else {
+                return new AudioFrame {
+                    timestamp = 0,
+                    frequency = input.frequency,
+                    channelCount = input.channelCount,
+                    samples = null
+                };
+            }
+        }
+    }
+
+    /// <summary>
+    /// Decodes Opus encoded audio. Use this as a filter for incoming client audio.
+    /// </summary>
+    public class OpusDecodeFilter : IAudioFilter {
+        Decoder decoder;
+        float[] outputBuffer;
+
+        public OpusDecodeFilter(Decoder decoder, int outputBufferLength = 48000) {
+            this.decoder = decoder;
+            outputBuffer = new float[outputBufferLength];
+        }
+
+        public AudioFrame Run(AudioFrame input) {
+            var decodeResult = decoder.Decode(input.samples, input.samples.Length, outputBuffer);
+            if(decodeResult > 0) {
+                float[] decoded = new float[decodeResult];
+                Array.Copy(outputBuffer, decoded, decoded.Length);
+                return new AudioFrame {
+                    timestamp = 0,
+                    frequency = input.frequency,
+                    channelCount = input.channelCount,
+                    samples = Utils.Bytes.FloatsToBytes(decoded)
+                };
+            }
+            else {
+                return new AudioFrame {
+                    timestamp = 0,
+                    frequency = input.frequency,
+                    channelCount = input.channelCount,
+                    samples = null
+                };
+            }
+        }
+    }
+}

package/Runtime/Impl/Filters/OpusFilter.cs.meta

@@ -0,0 +1,11 @@
+fileFormatVersion: 2
+guid: 31124d8beda4bc045b5ee5696140f0d8
+MonoImporter:
+  externalObjects: {}
+  serializedVersion: 2
+  defaultReferences: []
+  executionOrder: 0
+  icon: {instanceID: 0}
+  userData: 
+  assetBundleName: 
+  assetBundleVariant: 

package/Runtime/Impl/Filters.meta

@@ -0,0 +1,8 @@
+fileFormatVersion: 2
+guid: 1c33181b4e7cd6f4588b07afff029c77
+folderAsset: yes
+DefaultImporter:
+  externalObjects: {}
+  userData: 
+  assetBundleName: 
+  assetBundleVariant: 

package/Runtime/Impl/Inputs/UniMicInput.cs

@@ -0,0 +1,39 @@
+using System;
+
+using Adrenak.UniMic;
+
+using UnityEngine;
+
+namespace Adrenak.UniVoice.Inputs {
+    /// <summary>
+    /// An <see cref="IAudioInput"/> implementation based on UniMic.
+    /// For more on UniMic, visit https://www.github.com/adrenak/unimic
+    /// </summary>
+    public class UniMicInput : IAudioInput {
+        const string TAG = "UniMicInput";
+
+        public event Action<AudioFrame> OnFrameReady;
+
+        public Mic.Device Device { get; private set; }
+
+        public UniMicInput(Mic.Device device) {
+            Device = device;
+            device.OnFrameCollected += OnFrameCollected;
+        }
+
+        private void OnFrameCollected(int frequency, int channels, float[] samples) {
+            var frame = new AudioFrame {
+                timestamp = 0,
+                frequency = frequency,
+                channelCount = channels,
+                samples = Utils.Bytes.FloatsToBytes(samples)
+            };
+            OnFrameReady?.Invoke(frame);
+        }
+
+        public void Dispose() {
+            Device.OnFrameCollected -= OnFrameCollected;
+            Debug.unityLogger.Log(TAG, "Disposed");
+        }
+    }
+}

package/Runtime/Impl/Inputs/UniMicInput.cs.meta

@@ -0,0 +1,11 @@
+fileFormatVersion: 2
+guid: 6a98c26fdc5135842a8a651f2bd5f8e9
+MonoImporter:
+  externalObjects: {}
+  serializedVersion: 2
+  defaultReferences: []
+  executionOrder: 0
+  icon: {instanceID: 0}
+  userData: 
+  assetBundleName: 
+  assetBundleVariant: 

package/Runtime/Impl/Inputs.meta

@@ -0,0 +1,8 @@
+fileFormatVersion: 2
+guid: 569cfdf74a194604aa8913c641f27024
+folderAsset: yes
+DefaultImporter:
+  externalObjects: {}
+  userData: 
+  assetBundleName: 
+  assetBundleVariant: