Getatlas Ai6qy6oo3j

Evidently AI

Stay updated on the status of all your company's support tickets.

Customer portal

Understanding the distribution of your data is essential for building accurate and reliable machine learning models. Data distribution analysis helps you identify potential problems, such as outliers, skewness, and inconsistencies, that can affect model performance and prediction accuracy. Evidently AI provides powerful tools for visual analysis of your data distributions.

  Histograms help you visualize the frequency distribution of a single feature.  They show the frequency of data points within specific ranges, providing insights into the shape of the distribution, potential outliers, and the presence of skewness.

 Box plots offer a concise representation of the distribution of a feature, encompassing the median, quartiles, and potential outliers.  They highlight the spread of the data and help you identify any unusual values or changes in the data distribution compared to your reference data.

  Scatter plots are useful for visualizing the relationship between two features. They allow you to identify potential patterns, trends, and correlations between the features.

  CDFs show the probability that a feature's value is less than or equal to a given value. They are helpful for comparing the distributions of features in the reference and current data and identifying significant shifts.

 Kernel Density Estimation (KDE) plots provide a smoothed estimate of the probability density function of the data.  They offer a more continuous and smoother representation of the data's distribution compared to histograms.

 Q-Q plots compare the distribution of a feature in the reference and current data.  They help you visually assess whether the distributions are statistically similar or if there are significant differences. 

 Analyze the distributions of different features over time to detect potential data drift. This allows you to identify changes in the data patterns that could impact your model's accuracy.

Explore techniques for examining data distributions, outliers, and potential skewness in your datasets.

Data Distribution Analysis

Data quality is crucial for building reliable and accurate machine learning models.  While Evidently AI provides a range of built-in data quality checks, sometimes you need to create custom rules tailored to your specific domain and data requirements. Fortunately, Evidently AI allows you to define and apply your own custom data quality rules.

  Before defining custom rules, thoroughly understand your data, domain knowledge, and the specific requirements for data quality.  What are the unique aspects of your data that need to be validated?  What are the acceptable ranges or values for your features?

  Start by using the `DataDriftPreset` in your Evidently AI report to identify potential data quality issues. This provides a baseline for evaluating your data and understanding common inconsistencies or problems. 

  Evidently AI allows you to define your own custom metrics using the `Metric` class.  These metrics can be tailored to specific data quality checks you want to perform.

  You can create custom checks based on your specific requirements using the `Check` class. This allows you to define custom rules based on conditional statements or calculations.

  Within your custom checks, implement the logic to evaluate your data quality based on your defined rules. This might involve comparing values, calculating statistics, or performing custom calculations specific to your data.

Learn how to define your own custom data quality rules based on your specific domain requirements.

Customizing Data Quality Rules

Ensuring the quality of your data is crucial for building robust and reliable machine learning models. Data integrity checks help you identify and address potential issues in your data that can negatively impact model performance and create inaccurate predictions. Evidently AI provides powerful tools for conducting these crucial checks. 

  Evidently AI can detect missing values in your data, allowing you to understand the extent of the issue and identify features with missing data. This is important because missing data can lead to biased models and inaccurate predictions.

  Detecting duplicate rows in your data is important to ensure that your dataset is free from redundant entries. Duplicates can affect model training and may lead to inaccurate model estimates.

  Ensuring that all data points in a specific feature adhere to the correct data type is essential. This includes checks for numerical features (integers, floats), categorical features (text, strings), and timestamp data.  

  Validate that numerical data is stored in the correct format (e.g., dates, times, currencies). Invalid formats can cause errors during data processing or analysis.  

 Identifies extreme values in your data that may be errors or unusual events. Outliers can bias your model and affect its accuracy, especially when used in statistical modeling or as inputs to sensitive models. 

 Check that features should be unique, particularly in cases of identification numbers or unique identifiers.  

 Analyze distributions of your features, such as histograms or box plots. This helps you identify potentially problematic patterns. 

  Validate that values within features fall within expected ranges. This is important for numerical features where unrealistic values might indicate data errors.

Discover how to check for missing values, duplicate entries, and inconsistent data formats.

Data Quality Checks

Data Distribution Analysis

Customizing Data Quality Rules

Data Integrity Checks