Bootstrapping is a statistical technique that's used to draw an inference about the parameters of population based on the samples drawn from it with replacement and averaging these results out. In the event of sampling with replacement, samples are drawn one after another, and once one sample is drawn from the population, the population is replenished with the sampled data:

In the preceding diagram, there is a dataset that has got multiples components (A, B, C, D, E, F, G, H, and I). To start, we need to draw three samples of the same size. Let's draw Sample 1 randomly and say that the first element turned out to be A. However, before we draw the second element of Sample 1, A is returned to the dataset. A similar process takes place for the entire draw. This is called Sampling with Replacement. Hence, we have a chance of selecting the same item multiple times in a set. By following this process, we have drawn three samples, that is, Sample 1, Sample 2, and Sample 3.

When we take a step further down, which is determining the statistics (various metrics) on Sample 1, Sample 2, and Sample 3, we find out a mean or an average of all the statistics to infer something about the dataset (population). This entire process is called bootstrapping and the drawn samples are termed bootstrapped samples. This can be defined with the following equation:

Inference about the Dataset(Population) = Average(sample 1,sample 2,............,sample N)

If you look at the preceding diagram carefully, there might be a scenario wherein a few elements of the dataset haven't been picked or are not part of those three samples:

• Sample 1: (A, E, H, C)
• Sample 2: (F, G, A, C)
• Sample 3: (E, H, G, F)

Therefore, the elements that haven't been picked are B, D, and I. The samples that were not part of the drawn samples are called out-of-bag (OOB) samples.

Let's do a simple coding exercise to see how this can be done in Python:

1. Here, we will be using the sklearn and resample functions. Let's import the necessary libraries:

#importing Libraries
from sklearn.utils import resample

2. Next, create a dataset that we will need to sample:

dataset=[10,2

3. Now, we will extract a bootstrap sample with the help of the resample function:

0,30,40,50,60,70,80,90,100]

#using "resample" function generate a bootstrap sample
boot_samp = resample(dataset, replace=True, n_samples=5, random_state=1)

4. We will use list comprehension to extract an OOB sample:

#extracting OOB sample
OOB=[x for x in dataset if x not in boot_samp]

5. Now, let's print it:

print(boot_samp)

We get the following output:

[60, 90, 100, 60, 10]

We can see that there is a repetition of 60 in the sampling. This is due to sampling with replacement.

6. Next, we need to print the following code:

print(OOB)

We get the following output:

[20, 30, 40, 50, 70, 80]

By this end of this, we want to have a result that's as follows:

OOB = Dataset - Boot_Sample 

=[10,20,30,40,50,60,70,80,90,100] - [60,90,100,60,10]

=[20,30,40,50,70,80]

This is the same result we have got from the code.