# About Script to extract the api from [Unity](https://unity3d.com/) and save as json Web scraping on page https://docs.unity3d.com/ScriptReference/index.html # How to use - Install [**node.js**](https://nodejs.org/en/) - Open terminal - Go to **unity-scrapping** directory - `npm install` - `node --max-old-space-size=4096 main.js` - Unity api inside **api.json** # How it work It will navigate to unity website and read the HTML code to get what needed. ## Requires I will be reading this list | | | ------------------------ | | ![require](https://github.com/thiagola92/unity-scraping/raw/master/img/require.png) | The HTML format to find requires is `UnityEngine` #### Array of requires ```JSON { "text": "UnityEngine", "type": "require" }, { "text": "UnityEngine.Accessibility", "type": "require" }, { "text": "UnityEngine.Advertisements", "type": "require" }, ... ``` ## Objects Objects can be class/enumerations/... I will be reading this list again | | | ------------------------ | | ![object](https://github.com/thiagola92/unity-scraping/raw/master/img/object.png) | The HTML format to find objects is `VisionUtility` #### Array of objects ```JSON { "text": "VisionUtility", "descriptionMoreURL": "https://docs.unity3d.com/2019.1/Documentation/ScriptReference/Accessibility.VisionUtility.html", }, { "text": "Advertisement", "descriptionMoreURL": "https://docs.unity3d.com/2019.1/Documentation/ScriptReference/Advertisements.Advertisement.html", }, { "text": "DebugLevel", "descriptionMoreURL": "https://docs.unity3d.com/2019.1/Documentation/ScriptReference/Advertisements.Advertisement.DebugLevel.html", }, ... ``` ## Objects Information Using the link we can access the object page and get more information about it | | | ------------------------ | | ![objectInfo](https://github.com/thiagola92/unity-scraping/raw/master/img/objectinfo.png) | The HTML format to find the description is `

Description

A class containing methods to assist with accessibility for users with different vision capabilities.

` It going to be a short description, so it will only get the text until the `

` The HTML format to find the type is `

class in UnityEngine.Accessibility

` Is the first text after the `

` tag #### Update Array of objects ```JSON { "text": "VisionUtility", "descriptionMoreURL": "https://docs.unity3d.com/2019.1/Documentation/ScriptReference/Accessibility.VisionUtility.html", "description": "A class containing methods to assist with accessibility for users with different vision capabilities.", "type": "class" }, { "text": "Advertisement", "descriptionMoreURL": "https://docs.unity3d.com/2019.1/Documentation/ScriptReference/Advertisements.Advertisement.html", "description": "Unity Ads.", "type": "class" }, { "text": "DebugLevel", "descriptionMoreURL": "https://docs.unity3d.com/2019.1/Documentation/ScriptReference/Advertisements.Advertisement.DebugLevel.html", "description": "Player debug message level.", "type": "enumeration" }, ... ``` ## Static Properties We can find static properties searching the table | | | ------------------------ | | ![properties](https://github.com/thiagola92/unity-scraping/raw/master/img/properties.png) | The HTML format to find the propertie name, link, description `AllAreasArea mask constant that includes all NavMesh areas.` #### Array of Properties ```JSON { "text": "NavMesh.AllAreas", "description": "Area mask constant that includes all NavMesh areas.", "descriptionMoreURL": "https://docs.unity3d.com/2019.1/Documentation/ScriptReference/AI.NavMesh.AllAreas.html", "type": "property" }, { "text": "NavMesh.avoidancePredictionTime", "description": "Describes how far in the future the agents predict collisions for avoidance.", "descriptionMoreURL": "https://docs.unity3d.com/2019.1/Documentation/ScriptReference/AI.NavMesh-avoidancePredictionTime.html", "type": "property" }, { "text": "NavMesh.onPreUpdate", "description": "Set a function to be called before the NavMesh is updated during the frame update execution.", "descriptionMoreURL": "https://docs.unity3d.com/2019.1/Documentation/ScriptReference/AI.NavMesh-onPreUpdate.html", "type": "property" }, ... ``` ## Static Methods We can find static methods searching the table | | | ------------------------ | | ![methods](https://github.com/thiagola92/unity-scraping/raw/master/img/methods.png) | The HTML format to find the method name, link, description `AllAreasArea mask constant that includes all NavMesh areas.` #### Array of Methods ```JSON { "snippet": "NavMesh.AddLink", "description": "Adds a link to the NavMesh. The link is described by the NavMeshLinkData struct.", "descriptionMoreURL": "https://docs.unity3d.com/2019.1/Documentation/ScriptReference/AI.NavMesh.AddLink.html", "type": "method" }, { "snippet": "NavMesh.AddNavMeshData", "description": "Adds the specified NavMeshData to the game.", "descriptionMoreURL": "https://docs.unity3d.com/2019.1/Documentation/ScriptReference/AI.NavMesh.AddNavMeshData.html", "type": "method" }, ... ``` ## Signatures We can find the parameters from one signature because it's bold | | | ------------------------ | | ![signatures](https://github.com/thiagola92/unity-scraping/raw/master/img/signatures.png) | The HTML format to find parameters is `link` Obs: You have to ignore the first bold text because is not a parameter, the method name `AddLink` #### Array of Signatures ```JSON { "snippet": "NavMesh.AddLink(${1:link})", "description": "Adds a link to the NavMesh. The link is described by the NavMeshLinkData struct.", "descriptionMoreURL": "https://docs.unity3d.com/2019.1/Documentation/ScriptReference/AI.NavMesh.AddLink.html", "type": "method" }, { "snippet": "NavMesh.AddLink(${1:link}, ${2:position}, ${3:rotation})", "description": "Adds a link to the NavMesh. The link is described by the NavMeshLinkData struct.", "descriptionMoreURL": "https://docs.unity3d.com/2019.1/Documentation/ScriptReference/AI.NavMesh.AddLink.html", "type": "method" }, { "snippet": "NavMesh.AddNavMeshData(${1:navMeshData})", "description": "Adds the specified NavMeshData to the game.", "descriptionMoreURL": "https://docs.unity3d.com/2019.1/Documentation/ScriptReference/AI.NavMesh.AddNavMeshData.html", "type": "method" }, ... ``` # Problem Right now this code is synchronous, so it will take a long time to finish because it's opening one window at time First you open the API page * Requires * Doesn't need to open new window, so it's fast * Objects * Doesn't need to open new window, so it's fast * Objects information * **For each object it will open a new window** * Static properties * Doesn't need to open new window, so it's fast * Static Methods * Doesn't need to open new window, so it's fast * Signatures * **For each method it will open a new window**